[jira] [Created] (YARN-11161) Support getAttributesToNodes, getClusterNodeAttributes, getNodesToAttributes API's for Federation
fanshilun created YARN-11161: Summary: Support getAttributesToNodes, getClusterNodeAttributes, getNodesToAttributes API's for Federation Key: YARN-11161 URL: https://issues.apache.org/jira/browse/YARN-11161 Project: Hadoop YARN Issue Type: Sub-task Reporter: fanshilun -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11160) Support getResourceProfiles, getResourceProfile, getResourceTypeInfo API's for Federation
fanshilun created YARN-11160: Summary: Support getResourceProfiles, getResourceProfile, getResourceTypeInfo API's for Federation Key: YARN-11160 URL: https://issues.apache.org/jira/browse/YARN-11160 Project: Hadoop YARN Issue Type: Sub-task Reporter: fanshilun -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11159) Support failApplicationAttempt, updateApplicationPriority, updateApplicationTimeouts API's for Federation
fanshilun created YARN-11159: Summary: Support failApplicationAttempt, updateApplicationPriority, updateApplicationTimeouts API's for Federation Key: YARN-11159 URL: https://issues.apache.org/jira/browse/YARN-11159 Project: Hadoop YARN Issue Type: Sub-task Reporter: fanshilun -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11158) Support getDelegationToken, renewDelegationToken, cancelDelegationToken API's for Federation
fanshilun created YARN-11158: Summary: Support getDelegationToken, renewDelegationToken, cancelDelegationToken API's for Federation Key: YARN-11158 URL: https://issues.apache.org/jira/browse/YARN-11158 Project: Hadoop YARN Issue Type: Sub-task Reporter: fanshilun -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/ [May 15, 2022 4:16:06 PM] (noreply) YARN-11122. Support getClusterNodes API in FederationClientInterceptor (#4274) -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.mapred.TestLocalDistributedCacheManager hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/results-compile-javac-root.txt [340K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/results-javadoc-javadoc-root.txt [400K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-common.txt [48K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/871/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [68K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.3 (RC1)
+1 (binding) - Verified all checksums. - Verified all signatures. - Built from source, including native code on Linux. - Ran several examples successfully. Chris Nauroth On Mon, May 16, 2022 at 10:06 AM Chao Sun wrote: > +1 > > - Compiled from source > - Verified checksums & signatures > - Launched a pseudo HDFS cluster and ran some simple commands > - Ran full Spark tests with the RC > > Thanks Steve! > > Chao > > On Mon, May 16, 2022 at 2:19 AM Ayush Saxena wrote: > > > > +1, > > * Built from source. > > * Successful native build on Ubuntu 18.04 > > * Verified Checksums. > > > (CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz) > > * Verified Signature. > > * Successful RAT check > > * Ran basic HDFS shell commands. > > * Ran basic YARN shell commands. > > * Verified version in hadoop version command and UI > > * Ran some MR example Jobs. > > * Browsed UI(Namenode/Datanode/ResourceManager/NodeManager/HistoryServer) > > * Browsed the contents of Maven Artifacts. > > * Browsed the contents of the website. > > > > Thanx Steve for driving the release, Good Luck!!! > > > > -Ayush > > > > On Mon, 16 May 2022 at 08:20, Xiaoqiao He wrote: > > > > > +1(binding) > > > > > > * Verified signature and checksum of the source tarball. > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package > > > -DskipTests -Pnative -Pdist -Dtar`. > > > * Setup pseudo cluster with HDFS and YARN. > > > * Run simple FsShell - mkdir/put/get/mv/rm and check the result. > > > * Run example mr applications and check the result - Pi & wordcount. > > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager > etc. > > > > > > Thanks Steve for your work. > > > > > > - He Xiaoqiao > > > > > > On Mon, May 16, 2022 at 4:25 AM Viraj Jasani > wrote: > > > > > > > > +1 (non-binding) > > > > > > > > * Signature: ok > > > > * Checksum : ok > > > > * Rat check (1.8.0_301): ok > > > > - mvn clean apache-rat:check > > > > * Built from source (1.8.0_301): ok > > > > - mvn clean install -DskipTests > > > > * Built tar from source (1.8.0_301): ok > > > > - mvn clean package -Pdist -DskipTests -Dtar > -Dmaven.javadoc.skip=true > > > > > > > > HDFS, MapReduce and HBase (2.5) CRUD functional testing on > > > > pseudo-distributed mode looks good. > > > > > > > > > > > > On Wed, May 11, 2022 at 10:26 AM Steve Loughran > > > > > > > wrote: > > > > > > > > > I have put together a release candidate (RC1) for Hadoop 3.3.3 > > > > > > > > > > The RC is available at: > > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/ > > > > > > > > > > The git tag is release-3.3.3-RC1, commit d37586cbda3 > > > > > > > > > > The maven artifacts are staged at > > > > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1349/ > > > > > > > > > > You can find my public key at: > > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > > > Change log > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/CHANGELOG.md > > > > > > > > > > Release notes > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/RELEASENOTES.md > > > > > > > > > > There's a very small number of changes, primarily critical > > > code/packaging > > > > > issues and security fixes. > > > > > > > > > > * The critical fixes which shipped in the 3.2.3 release. > > > > > * CVEs in our code and dependencies > > > > > * Shaded client packaging issues. > > > > > * A switch from log4j to reload4j > > > > > > > > > > reload4j is an active fork of the log4j 1.17 library with the > classes > > > > > which contain CVEs removed. Even though hadoop never used those > > > classes, > > > > > they regularly raised alerts on security scans and concen from > users. > > > > > Switching to the forked project allows us to ship a secure logging > > > > > framework. It will complicate the builds of downstream > > > > > maven/ivy/gradle projects which exclude our log4j artifacts, as > they > > > > > need to cut the new dependency instead/as well. > > > > > > > > > > See the release notes for details. > > > > > > > > > > This is the second release attempt. It is the same git commit as > > > before, > > > > > but > > > > > fully recompiled with another republish to maven staging, which > has bee > > > > > verified by building spark, as well as a minimal test project. > > > > > > > > > > Please try the release and vote. The vote will run for 5 days. > > > > > > > > > > -Steve > > > > > > > > > > > - > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > - > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org >
Re: [VOTE] Release Apache Hadoop 3.3.3 (RC1)
+1 - Compiled from source - Verified checksums & signatures - Launched a pseudo HDFS cluster and ran some simple commands - Ran full Spark tests with the RC Thanks Steve! Chao On Mon, May 16, 2022 at 2:19 AM Ayush Saxena wrote: > > +1, > * Built from source. > * Successful native build on Ubuntu 18.04 > * Verified Checksums. > (CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz) > * Verified Signature. > * Successful RAT check > * Ran basic HDFS shell commands. > * Ran basic YARN shell commands. > * Verified version in hadoop version command and UI > * Ran some MR example Jobs. > * Browsed UI(Namenode/Datanode/ResourceManager/NodeManager/HistoryServer) > * Browsed the contents of Maven Artifacts. > * Browsed the contents of the website. > > Thanx Steve for driving the release, Good Luck!!! > > -Ayush > > On Mon, 16 May 2022 at 08:20, Xiaoqiao He wrote: > > > +1(binding) > > > > * Verified signature and checksum of the source tarball. > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package > > -DskipTests -Pnative -Pdist -Dtar`. > > * Setup pseudo cluster with HDFS and YARN. > > * Run simple FsShell - mkdir/put/get/mv/rm and check the result. > > * Run example mr applications and check the result - Pi & wordcount. > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc. > > > > Thanks Steve for your work. > > > > - He Xiaoqiao > > > > On Mon, May 16, 2022 at 4:25 AM Viraj Jasani wrote: > > > > > > +1 (non-binding) > > > > > > * Signature: ok > > > * Checksum : ok > > > * Rat check (1.8.0_301): ok > > > - mvn clean apache-rat:check > > > * Built from source (1.8.0_301): ok > > > - mvn clean install -DskipTests > > > * Built tar from source (1.8.0_301): ok > > > - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true > > > > > > HDFS, MapReduce and HBase (2.5) CRUD functional testing on > > > pseudo-distributed mode looks good. > > > > > > > > > On Wed, May 11, 2022 at 10:26 AM Steve Loughran > > > > > wrote: > > > > > > > I have put together a release candidate (RC1) for Hadoop 3.3.3 > > > > > > > > The RC is available at: > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/ > > > > > > > > The git tag is release-3.3.3-RC1, commit d37586cbda3 > > > > > > > > The maven artifacts are staged at > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1349/ > > > > > > > > You can find my public key at: > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > Change log > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/CHANGELOG.md > > > > > > > > Release notes > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/RELEASENOTES.md > > > > > > > > There's a very small number of changes, primarily critical > > code/packaging > > > > issues and security fixes. > > > > > > > > * The critical fixes which shipped in the 3.2.3 release. > > > > * CVEs in our code and dependencies > > > > * Shaded client packaging issues. > > > > * A switch from log4j to reload4j > > > > > > > > reload4j is an active fork of the log4j 1.17 library with the classes > > > > which contain CVEs removed. Even though hadoop never used those > > classes, > > > > they regularly raised alerts on security scans and concen from users. > > > > Switching to the forked project allows us to ship a secure logging > > > > framework. It will complicate the builds of downstream > > > > maven/ivy/gradle projects which exclude our log4j artifacts, as they > > > > need to cut the new dependency instead/as well. > > > > > > > > See the release notes for details. > > > > > > > > This is the second release attempt. It is the same git commit as > > before, > > > > but > > > > fully recompiled with another republish to maven staging, which has bee > > > > verified by building spark, as well as a minimal test project. > > > > > > > > Please try the release and vote. The vote will run for 5 days. > > > > > > > > -Steve > > > > > > > > - > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11133) YarnClient gets the wrong EffectiveMinCapacity value
[
https://issues.apache.org/jira/browse/YARN-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Boyina resolved YARN-11133.
---
Fix Version/s: 3.4.0
Resolution: Fixed
> YarnClient gets the wrong EffectiveMinCapacity value
>
>
> Key: YARN-11133
> URL: https://issues.apache.org/jira/browse/YARN-11133
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 3.2.3, 3.3.2
>Reporter: Zilong Zhu
>Assignee: Zilong Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> It calls the QueueConfigurations#getEffectiveMinCapacity to get the wrong
> value when I use the YarnClient. I found some bugs with
> QueueConfigurationsPBImpl#mergeLocalToBuilder.
> {code:java}
> private void mergeLocalToBuilder() {
> if (this.effMinResource != null) {
> builder
> .setEffectiveMinCapacity(convertToProtoFormat(this.effMinResource));
> }
> if (this.effMaxResource != null) {
> builder
> .setEffectiveMaxCapacity(convertToProtoFormat(this.effMaxResource));
> }
> if (this.configuredMinResource != null) {
> builder.setEffectiveMinCapacity(
> convertToProtoFormat(this.configuredMinResource));
> }
> if (this.configuredMaxResource != null) {
> builder.setEffectiveMaxCapacity(
> convertToProtoFormat(this.configuredMaxResource));
> }
> } {code}
> configuredMinResource was incorrectly assigned to effMinResource. This causes
> the real effMinResource to be overwritten and configuredMinResource is null.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [DISCUSS] Hadoop 2.10.2 release
Hi committers, I created branch-2.10.2 and bumped the version of branch-2.10 to 2.10.3-SNAPSHOT. Please set the "Fix Version/s:" of JIRA to 2.10.3 if you commit the fix to branch-2.10 without cherry-picking to branch-2.10.2. Regards, Masatake Iwasaki On 2022/05/16 23:16, Masatake Iwasaki wrote: Thanks, Akira and Xiaoqiao for positive feedback. I'm going to cut branch-2.10.2 shortly. Masatake Iwasaki On 2022/05/12 11:38, Xiaoqiao He wrote: Thanks Masatake and Akira for bringing this release up. +1 for 2.10.2 release. Sorry for the late response. Regards, - He Xiaoqiao On Mon, Mar 14, 2022 at 1:17 PM Akira Ajisaka wrote: Thank you Masatake. +1 to release 2.10.2. On Fri, Mar 4, 2022 at 6:52 PM Masatake Iwasaki wrote: Hi team, There are over 170 fixed issues in branch-2.10 after the release of 2.10.1[1]. Given that there is still a need for 2.10.x, I would like to release 2.10.2 after HADOOP-18088[2] (migration to reload4j) is merged. I volunteer to take a release manager role as done for 2.10.1. Maybe we can declare EOL of branch-2.10 after the release, it should be discussed in another thread. [1] https://issues.apache.org/jira/issues/?jql=project%20in%20(YARN%2C%20HDFS%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.10.2 [2] https://issues.apache.org/jira/browse/HADOOP-18088 Thanks, Masatake Iwasaki - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11092) Upgrade jquery ui to 1.13.1
[ https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved YARN-11092. -- Fix Version/s: 3.4.0 3.2.4 3.3.4 Resolution: Fixed Committed to trunk, branch-3.3, and branch-3.2. Thank you [~dmmkr] for the report and thank you [~groot] for your contribution. > Upgrade jquery ui to 1.13.1 > --- > > Key: YARN-11092 > URL: https://issues.apache.org/jira/browse/YARN-11092 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.4 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The current jquery-ui version used(1.12.1) in the trunk has the following > vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to > upgrade to at least 1.13.0. > > Also currently for the UI2 we are using the shims repo which is not being > maintained as per the discussion > [https://github.com/components/jqueryui/issues/70] , so if possible we should > move to the main jquery repo [https://github.com/jquery/jquery-ui] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [DISCUSS] Hadoop 2.10.2 release
Thanks, Akira and Xiaoqiao for positive feedback. I'm going to cut branch-2.10.2 shortly. Masatake Iwasaki On 2022/05/12 11:38, Xiaoqiao He wrote: Thanks Masatake and Akira for bringing this release up. +1 for 2.10.2 release. Sorry for the late response. Regards, - He Xiaoqiao On Mon, Mar 14, 2022 at 1:17 PM Akira Ajisaka wrote: Thank you Masatake. +1 to release 2.10.2. On Fri, Mar 4, 2022 at 6:52 PM Masatake Iwasaki wrote: Hi team, There are over 170 fixed issues in branch-2.10 after the release of 2.10.1[1]. Given that there is still a need for 2.10.x, I would like to release 2.10.2 after HADOOP-18088[2] (migration to reload4j) is merged. I volunteer to take a release manager role as done for 2.10.1. Maybe we can declare EOL of branch-2.10 after the release, it should be discussed in another thread. [1] https://issues.apache.org/jira/issues/?jql=project%20in%20(YARN%2C%20HDFS%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.10.2 [2] https://issues.apache.org/jira/browse/HADOOP-18088 Thanks, Masatake Iwasaki - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11157) Root queue state is not indicated in Capacity Scheduler scheduler response
András Győri created YARN-11157: --- Summary: Root queue state is not indicated in Capacity Scheduler scheduler response Key: YARN-11157 URL: https://issues.apache.org/jira/browse/YARN-11157 Project: Hadoop YARN Issue Type: Improvement Components: capacity scheduler Reporter: András Győri Assignee: András Győri Root queue state can be changed, but the /scheduler endpoint does not include the actual state of the root queue. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11156) Effective capacity not correct
chenwencan created YARN-11156:
-
Summary: Effective capacity not correct
Key: YARN-11156
URL: https://issues.apache.org/jira/browse/YARN-11156
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler
Affects Versions: 3.3.2, 3.3.1
Reporter: chenwencan
Attachments: effective capacity.jpg
when queue capacity type is absolute，if cluster capacity is less than queue
configured capacity，but some resource type in cluster capacity is lager than
queue configured capacity，the queue effective capacity not correct（used cluster
capacity actural）
for example
{code:java}
cluster capacity: 12288MB / 6vcores
root.test configured capacity: 7168MB / 8vcores {code}
after run normal:
{code:java}
root.test configured capacity: 12288MB / 6vcores
{code}
but I think it should be 7168MB / 6vcores
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.tools.TestDistCpSystem hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.resourceestimator.service.TestResourceEstimatorService cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/diff-compile-javac-root.txt [476K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-mvnsite-root.txt [560K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [216K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [428K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [116K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt [32K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/663/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K]
Re: [VOTE] Release Apache Hadoop 3.3.3 (RC1)
+1, * Built from source. * Successful native build on Ubuntu 18.04 * Verified Checksums. (CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz) * Verified Signature. * Successful RAT check * Ran basic HDFS shell commands. * Ran basic YARN shell commands. * Verified version in hadoop version command and UI * Ran some MR example Jobs. * Browsed UI(Namenode/Datanode/ResourceManager/NodeManager/HistoryServer) * Browsed the contents of Maven Artifacts. * Browsed the contents of the website. Thanx Steve for driving the release, Good Luck!!! -Ayush On Mon, 16 May 2022 at 08:20, Xiaoqiao He wrote: > +1(binding) > > * Verified signature and checksum of the source tarball. > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package > -DskipTests -Pnative -Pdist -Dtar`. > * Setup pseudo cluster with HDFS and YARN. > * Run simple FsShell - mkdir/put/get/mv/rm and check the result. > * Run example mr applications and check the result - Pi & wordcount. > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc. > > Thanks Steve for your work. > > - He Xiaoqiao > > On Mon, May 16, 2022 at 4:25 AM Viraj Jasani wrote: > > > > +1 (non-binding) > > > > * Signature: ok > > * Checksum : ok > > * Rat check (1.8.0_301): ok > > - mvn clean apache-rat:check > > * Built from source (1.8.0_301): ok > > - mvn clean install -DskipTests > > * Built tar from source (1.8.0_301): ok > > - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true > > > > HDFS, MapReduce and HBase (2.5) CRUD functional testing on > > pseudo-distributed mode looks good. > > > > > > On Wed, May 11, 2022 at 10:26 AM Steve Loughran > > > wrote: > > > > > I have put together a release candidate (RC1) for Hadoop 3.3.3 > > > > > > The RC is available at: > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/ > > > > > > The git tag is release-3.3.3-RC1, commit d37586cbda3 > > > > > > The maven artifacts are staged at > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1349/ > > > > > > You can find my public key at: > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > Change log > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/CHANGELOG.md > > > > > > Release notes > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/RELEASENOTES.md > > > > > > There's a very small number of changes, primarily critical > code/packaging > > > issues and security fixes. > > > > > > * The critical fixes which shipped in the 3.2.3 release. > > > * CVEs in our code and dependencies > > > * Shaded client packaging issues. > > > * A switch from log4j to reload4j > > > > > > reload4j is an active fork of the log4j 1.17 library with the classes > > > which contain CVEs removed. Even though hadoop never used those > classes, > > > they regularly raised alerts on security scans and concen from users. > > > Switching to the forked project allows us to ship a secure logging > > > framework. It will complicate the builds of downstream > > > maven/ivy/gradle projects which exclude our log4j artifacts, as they > > > need to cut the new dependency instead/as well. > > > > > > See the release notes for details. > > > > > > This is the second release attempt. It is the same git commit as > before, > > > but > > > fully recompiled with another republish to maven staging, which has bee > > > verified by building spark, as well as a minimal test project. > > > > > > Please try the release and vote. The vote will run for 5 days. > > > > > > -Steve > > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
[jira] [Created] (YARN-11155) ATS v1.5 doesn't work with JWTRedirectAuthenticationHandler
KWON BYUNGCHANG created YARN-11155:
--
Summary: ATS v1.5 doesn't work with
JWTRedirectAuthenticationHandler
Key: YARN-11155
URL: https://issues.apache.org/jira/browse/YARN-11155
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Affects Versions: 3.3.2, 3.1.2
Reporter: KWON BYUNGCHANG
When ATS is configured with JWTRedirectAuthenticationHandler for KnoxSSO, In
ATS, Delegation Token operation does not work.
In this situation, All hadoop web daemon use JWTRedirectAuthenticationHandler
for KnoxSSO. But ATS should be use kerberos auth handler. Tez job users should
login to kerberos for spnego auth for tez-ui access in own local pc. It is very
inconvenient.
Expected result (use JWTRedirectAuthenticationHandler)
{code:java}
curl -s -u: --negotiate
"https://ats.host.com:8190/ws/v1/timeline/?op=GETDELEGATIONTOKEN&=rm%2Frm1.host.com%40EXAMPLE.ORG;
{
"Token": {
"urlString": "KAbnVtLWFkbWm8EsIAZVElNfREVMRUTl9UT0tFTgA"
}
}
{code}
Wrong result (use JWTRedirectAuthenticationHandler)
{code:java}
curl -s -u: --negotiate
"https://ats.host.com:8190/ws/v1/timeline/?op=GETDELEGATIONTOKEN&=rm%2Frm1.host.com%40EXAMPLE.ORG;
{
"About": "Timeline API",
"hadoop-build-version": "3.1.2 from
7c62584effd9a5aa4b90d22dbf8d8eb2bca03feb by irteam source checksum
444e3aaa7feb4f8f73c3c3a71dbdd38",
"hadoop-version": "3.1.2",
"hadoop-version-built-on": "2022-04-08T03:45Z",
"timeline-service-build-version": "3.1.2-49 from
7c62584effd9a5aa4b90d22dbf8d8eb2bca03feb by users source checksum
7594ee7186b86eeccfc787d139ee8b",
"timeline-service-version": "3.1.2",
"timeline-service-version-built-on": "2022-04-08T03:49Z"
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11154) Make router support proxy server.
zhengchenyu created YARN-11154: -- Summary: Make router support proxy server. Key: YARN-11154 URL: https://issues.apache.org/jira/browse/YARN-11154 Project: Hadoop YARN Issue Type: Improvement Components: yarn Affects Versions: 3.2.1 Reporter: zhengchenyu Assignee: zhengchenyu Fix For: 3.4.0 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11153) Make proxy server support yarn federation.
zhengchenyu created YARN-11153: -- Summary: Make proxy server support yarn federation. Key: YARN-11153 URL: https://issues.apache.org/jira/browse/YARN-11153 Project: Hadoop YARN Issue Type: Improvement Components: yarn Affects Versions: 3.2.1 Reporter: zhengchenyu Assignee: zhengchenyu Fix For: 3.4.0 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11152) QueueMetrics is leaking memory when creating a new queue during reinitialisation
András Győri created YARN-11152: --- Summary: QueueMetrics is leaking memory when creating a new queue during reinitialisation Key: YARN-11152 URL: https://issues.apache.org/jira/browse/YARN-11152 Project: Hadoop YARN Issue Type: Bug Components: capacity scheduler Reporter: András Győri Assignee: András Győri Capacity Scheduler handles reinitialisation by reparsing the entire queue hierarchy, then reinitialising the old queue hierarchy by taking the newly parsed queues into account. After this, the newly parsed queues are discarded and they are GCed. However, with the introduction of YARN-6492, we are storing a parent queue in QueueMetrics, which is problematic, because at that point, the parent queue could still point to a parent reference, that is a newly parsed parent queue (which should be discarded after the reinitialisation). Due to this fact, QueueMetrics could contain parents members of an entirely different queue hierarchy than the current hierarchy in use. It could lead to subtle problems as well as memory leak, because one parent reference will keep the whole queue hierarchy alive. This problem arised when we programatically added one queue after an other via the mutation API, thus keeping alive hundreds of queue hierarchies at the same time, crippling the GC and the whole RM. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
