Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/228/ No changes -1 overall The following subsystems voted -1: docker Powered by Apache Yetushttps://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/ [Mar 3, 2021 1:34:02 PM] (noreply) HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) [Mar 3, 2021 4:44:30 PM] (Peter Bacsko) YARN-10655. Limit queue creation depth relative to its first static parent. Contributed by Andras Gyori. [Mar 4, 2021 5:55:37 AM] (Akira Ajisaka) HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. [Mar 4, 2021 9:22:58 AM] (noreply) YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie [Mar 4, 2021 11:23:11 AM] (Peter Bacsko) YARN-10623. Capacity scheduler should support refresh queue automatically by a thread policy. Contributed by Qi Zhu. [Mar 4, 2021 4:18:35 PM] (Peter Bacsko) YARN-10532. Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used. Contributed by Qi Zhu. -1 overall The following subsystems voted -1: blanks mvnsite pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.TestDecommissionWithStriped hadoop.yarn.client.TestRMFailoverProxyProvider hadoop.yarn.client.TestNoHaRMFailoverProxyProvider hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor hadoop.yarn.server.timelineservice.documentstore.TestDocumentStoreTimelineReaderImpl hadoop.yarn.server.timelineservice.documentstore.TestDocumentStoreCollectionCreator hadoop.yarn.server.timelineservice.documentstore.writer.cosmosdb.TestCosmosDBDocumentStoreWriter hadoop.yarn.server.timelineservice.documentstore.reader.cosmosdb.TestCosmosDBDocumentStoreReader hadoop.yarn.server.timelineservice.documentstore.TestDocumentStoreTimelineWriterImpl hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.TestDynamometerInfra cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/results-compile-cc-root.txt [116K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/results-compile-javac-root.txt [392K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/results-checkstyle-root.txt [16M] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/patch-mvnsite-root.txt [496K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/results-javadoc-javadoc-root.txt [304K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [332K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/133/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-ya
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/ [Mar 4, 2021 5:55:37 AM] (Akira Ajisaka) HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. [Mar 4, 2021 9:22:58 AM] (noreply) YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie [Mar 4, 2021 11:23:11 AM] (Peter Bacsko) YARN-10623. Capacity scheduler should support refresh queue automatically by a thread policy. Contributed by Qi Zhu. [Mar 4, 2021 4:18:35 PM] (Peter Bacsko) YARN-10532. Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used. Contributed by Qi Zhu. -1 overall The following subsystems voted -1: blanks pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks hadoop.yarn.client.api.impl.TestAMRMClient hadoop.tools.dynamometer.TestDynamometerInfra hadoop.tools.dynamometer.TestDynamometerInfra cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/results-compile-cc-root.txt [116K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/results-compile-javac-root.txt [368K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/results-checkstyle-root.txt [16M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/results-javadoc-javadoc-root.txt [1.1M] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [328K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer_hadoop-dynamometer-infra.txt [8.0K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/437/artifact/out/patch-unit-hadoop-tools_hadoop-dynamometer.txt [24K] Powered by Apache Yetus 0.13.0 https://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8786) LinuxContainerExecutor fails sporadically in create_local_dirs
[
https://issues.apache.org/jira/browse/YARN-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko resolved YARN-8786.
Resolution: Fixed
> LinuxContainerExecutor fails sporadically in create_local_dirs
> --
>
> Key: YARN-8786
> URL: https://issues.apache.org/jira/browse/YARN-8786
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jon Bender
>Priority: Major
>
> We started using CGroups with LinuxContainerExecutor recently, running Apache
> Hadoop 3.0.0. Occasionally (once out of many millions of tasks) a yarn
> container will fail with a message like the following:
> {code:java}
> [2018-09-02 23:48:02.458691] 18/09/02 23:48:02 INFO container.ContainerImpl:
> Container container_1530684675517_516620_01_020846 transitioned from
> SCHEDULED to RUNNING
> [2018-09-02 23:48:02.458874] 18/09/02 23:48:02 INFO
> monitor.ContainersMonitorImpl: Starting resource-monitoring for
> container_1530684675517_516620_01_020846
> [2018-09-02 23:48:02.506114] 18/09/02 23:48:02 WARN
> privileged.PrivilegedOperationExecutor: Shell execution returned exit code:
> 35. Privileged Execution Operation Stderr:
> [2018-09-02 23:48:02.506159] Could not create container dirsCould not create
> local files and directories
> [2018-09-02 23:48:02.506220]
> [2018-09-02 23:48:02.506238] Stdout: main : command provided 1
> [2018-09-02 23:48:02.506258] main : run as user is nobody
> [2018-09-02 23:48:02.506282] main : requested yarn user is root
> [2018-09-02 23:48:02.506294] Getting exit code file...
> [2018-09-02 23:48:02.506307] Creating script paths...
> [2018-09-02 23:48:02.506330] Writing pid file...
> [2018-09-02 23:48:02.506366] Writing to tmp file
> /path/to/hadoop/yarn/local/nmPrivate/application_1530684675517_516620/container_1530684675517_516620_01_020846/container_1530684675517_516620_01_020846.pid.tmp
> [2018-09-02 23:48:02.506389] Writing to cgroup task files...
> [2018-09-02 23:48:02.506402] Creating local dirs...
> [2018-09-02 23:48:02.506414] Getting exit code file...
> [2018-09-02 23:48:02.506435] Creating script paths...
> {code}
> Looking at the container executor source it's traceable to errors here:
> [https://github.com/apache/hadoop/blob/release-3.0.0-RC1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c#L1604]
> And ultimately to
> [https://github.com/apache/hadoop/blob/release-3.0.0-RC1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c#L672]
> The root failure seems to be in the underlying mkdir call, but that exit code
> / errno is swallowed so we don't have more details. We tend to see this when
> many containers start at the same time for the same application on a host,
> and suspect it may be related to some race conditions around those shared
> directories between containers for the same application.
> For example, this is a typical pattern in the audit logs:
> {code:java}
> [2018-09-07 17:16:38.447654] 18/09/07 17:16:38 INFO
> nodemanager.NMAuditLogger: USER=root IP=<> Container Request
> TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_1530684675517_559126
> CONTAINERID=container_1530684675517_559126_01_012871
> [2018-09-07 17:16:38.492298] 18/09/07 17:16:38 INFO
> nodemanager.NMAuditLogger: USER=root IP=<> Container Request
> TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_1530684675517_559126
> CONTAINERID=container_1530684675517_559126_01_012870
> [2018-09-07 17:16:38.614044] 18/09/07 17:16:38 WARN
> nodemanager.NMAuditLogger: USER=root OPERATION=Container Finished -
> Failed TARGET=ContainerImplRESULT=FAILURE DESCRIPTION=Container failed
> with state: EXITED_WITH_FAILUREAPPID=application_1530684675517_559126
> CONTAINERID=container_1530684675517_559126_01_012871
> {code}
> Two containers for the same application starting in quick succession followed
> by the EXITED_WITH_FAILURE step (exit code 35).
> We plan to upgrade to 3.1.x soon but I don't expect this to be fixed by this,
> the only major JIRAs that affected the executor since 3.0.0 seem unrelated
> ([https://github.com/apache/hadoop/commit/bc285da107bb84a3c60c5224369d7398a41db2d8]
> and
> [https://github.com/apache/hadoop/commit/a82be7754d74f4d16b206427b91e700bb5f44d56])
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10676) Improve code quality in TestTimelineAuthenticationFilterForV1
Szilard Nemeth created YARN-10676: - Summary: Improve code quality in TestTimelineAuthenticationFilterForV1 Key: YARN-10676 URL: https://issues.apache.org/jira/browse/YARN-10676 Project: Hadoop YARN Issue Type: Bug Reporter: Szilard Nemeth Assignee: Szilard Nemeth -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10643) Fix the race condition introduced by YARN-8995.
[ https://issues.apache.org/jira/browse/YARN-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko resolved YARN-10643. - Resolution: Duplicate > Fix the race condition introduced by YARN-8995. > --- > > Key: YARN-10643 > URL: https://issues.apache.org/jira/browse/YARN-10643 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.0, 3.2.1 >Reporter: Qi Zhu >Assignee: zhengchenyu >Priority: Critical > Attachments: YARN-10643.001.patch > > > The race condition introduced by -YARN-8995.- > The problem has been raised in YARN-10221 > also in YARN-10642. > I think we should fix it in a hurry. > I will help fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10675) Consolidate YARN-10672 and YARN-10447
Szilard Nemeth created YARN-10675:
-
Summary: Consolidate YARN-10672 and YARN-10447
Key: YARN-10675
URL: https://issues.apache.org/jira/browse/YARN-10675
Project: Hadoop YARN
Issue Type: Bug
Reporter: Szilard Nemeth
Assignee: Szilard Nemeth
Let's consolidate the solution applied for YARN-10672 and apply it to the code
changes introduced with YARN-10447.
Quoting [~pbacsko]:
{quote}
The solution is much straightforward than mine in YARN-10447. Actually we might
consider applying this to TestLeafQueue with undoing my changes, because that's
more complicated (I had no patience to go deeper with Mockito internal
behavior, I just thought well, disable that thread and that's enough).
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10674) fs2cs: should support auto created queue deletion.
Qi Zhu created YARN-10674: - Summary: fs2cs: should support auto created queue deletion. Key: YARN-10674 URL: https://issues.apache.org/jira/browse/YARN-10674 Project: Hadoop YARN Issue Type: Sub-task Reporter: Qi Zhu Assignee: Qi Zhu -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10673) Fix the spelling errors in TestCapacitySchedulerWeightMode about allocation.
Qi Zhu created YARN-10673: - Summary: Fix the spelling errors in TestCapacitySchedulerWeightMode about allocation. Key: YARN-10673 URL: https://issues.apache.org/jira/browse/YARN-10673 Project: Hadoop YARN Issue Type: Sub-task Reporter: Qi Zhu Assignee: Qi Zhu -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
