Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4
+1 (non-binding) as well - did a full CI run from Spark side with the RC and it looks good - verified checksums and signatures - started a docker-based single node cluster and tested with basic HDFS commands - verified NN webui and it looks good On Wed, Dec 16, 2020 at 3:54 PM Wei-Chiu Chuang wrote: > Forgot to mention I am +1 (binding) too based on the fact there are no > blocker issues and RC3 and RC4 diff. > > On Mon, Dec 14, 2020 at 1:42 PM Wei-Chiu Chuang > wrote: > > > Diff between RC3 and RC4: > > > > MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails > (#2136) > > HDFS-13404. Addendum: RBF: > > TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. > > Contributed by Takanobu Asanuma. > > HADOOP-16080. hadoop-aws does not work with hadoop-client-api (#2510). > > Contributed by Chao Sun > > HADOOP-15775. [JDK9] Add missing javax.activation-api dependency. > > Contributed by Akira Ajisaka. > > HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block > > error. Contributed by HuangTao. > > HDFS-15708. TestURLConnectionFactory fails by NoClassDefFoundError in > > branch-3.3 and branch-3.2 (#2517) > > HADOOP-17389. KMS should log full UGI principal. (#2476) > > HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by > > Ahmed Hussein and Daryn Sharp > > HDFS-15709. Socket file descriptor leak in > > StripedBlockChecksumReconstructor. (#2518) > > > > > > On Wed, Dec 9, 2020 at 9:01 AM Xiaoqiao He > wrote: > > > >> Hi folks, > >> > >> The release candidate (RC4) for Hadoop-3.2.2 is available now. > >> There are 10 commits[1] differences between RC4 and RC3[2]. > >> > >> The RC4 is available at: > >> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC4 > >> The RC4 tag in github is here: > >> https://github.com/apache/hadoop/tree/release-3.2.2-RC4 > >> The maven artifacts are staged at: > >> https://repository.apache.org/content/repositories/orgapachehadoop-1296 > >> > >> You can find my public key at: > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or > >> https://people.apache.org/keys/committer/hexiaoqiao.asc directly. > >> > >> Please try the release and vote. > >> > >> I have done a simple testing with my pseudo cluster. > >> * Verify gpg sign and md5sum. > >> * Setup pseudo cluster with HDFS and YARN. > >> * Run simple FsShell - mkdir/put/get/rename/mv. > >> * Submit example mr job and check the result - Pi/wordcount. > >> * Check the web UI of NameNode/DataNode/Resourcemanager/NodeManager. > >> My +1 to start. > >> > >> Thanks, > >> He Xiaoqiao > >> > >> [1] > >> > >> > https://github.com/apache/hadoop/compare/release-3.2.2-RC3...release-3.2.2-RC4 > >> [2] > >> > >> > https://lists.apache.org/thread.html/rfb74c3a5d4f223c5804d8ee622829263740cd8701c8f3fc8b6f970af%40%3Chdfs-dev.hadoop.apache.org%3E > >> [3] > >> > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948 > >> > > >
Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4
Forgot to mention I am +1 (binding) too based on the fact there are no blocker issues and RC3 and RC4 diff. On Mon, Dec 14, 2020 at 1:42 PM Wei-Chiu Chuang wrote: > Diff between RC3 and RC4: > > MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails (#2136) > HDFS-13404. Addendum: RBF: > TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. > Contributed by Takanobu Asanuma. > HADOOP-16080. hadoop-aws does not work with hadoop-client-api (#2510). > Contributed by Chao Sun > HADOOP-15775. [JDK9] Add missing javax.activation-api dependency. > Contributed by Akira Ajisaka. > HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block > error. Contributed by HuangTao. > HDFS-15708. TestURLConnectionFactory fails by NoClassDefFoundError in > branch-3.3 and branch-3.2 (#2517) > HADOOP-17389. KMS should log full UGI principal. (#2476) > HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by > Ahmed Hussein and Daryn Sharp > HDFS-15709. Socket file descriptor leak in > StripedBlockChecksumReconstructor. (#2518) > > > On Wed, Dec 9, 2020 at 9:01 AM Xiaoqiao He wrote: > >> Hi folks, >> >> The release candidate (RC4) for Hadoop-3.2.2 is available now. >> There are 10 commits[1] differences between RC4 and RC3[2]. >> >> The RC4 is available at: >> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC4 >> The RC4 tag in github is here: >> https://github.com/apache/hadoop/tree/release-3.2.2-RC4 >> The maven artifacts are staged at: >> https://repository.apache.org/content/repositories/orgapachehadoop-1296 >> >> You can find my public key at: >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or >> https://people.apache.org/keys/committer/hexiaoqiao.asc directly. >> >> Please try the release and vote. >> >> I have done a simple testing with my pseudo cluster. >> * Verify gpg sign and md5sum. >> * Setup pseudo cluster with HDFS and YARN. >> * Run simple FsShell - mkdir/put/get/rename/mv. >> * Submit example mr job and check the result - Pi/wordcount. >> * Check the web UI of NameNode/DataNode/Resourcemanager/NodeManager. >> My +1 to start. >> >> Thanks, >> He Xiaoqiao >> >> [1] >> >> https://github.com/apache/hadoop/compare/release-3.2.2-RC3...release-3.2.2-RC4 >> [2] >> >> https://lists.apache.org/thread.html/rfb74c3a5d4f223c5804d8ee622829263740cd8701c8f3fc8b6f970af%40%3Chdfs-dev.hadoop.apache.org%3E >> [3] >> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948 >> >
[jira] [Created] (YARN-10536) Client in distributedShell swallows interrupt exceptions
Ahmed Hussein created YARN-10536:
Summary: Client in distributedShell swallows interrupt exceptions
Key: YARN-10536
URL: https://issues.apache.org/jira/browse/YARN-10536
Project: Hadoop YARN
Issue Type: Bug
Components: client, distributed-shell
Reporter: Ahmed Hussein
Assignee: Ahmed Hussein
In {{applications.distributedshell.Client}} , the method {{monitorApplication}}
loops waiting for the following conditions:
* Application fails: reaches {{YarnApplicationState.KILLED}}, or
{{YarnApplicationState.FAILED}}
* Application succeeds: {{FinalApplicationStatus.SUCCEEDED}} or
{{YarnApplicationState.FINISHED}}
* the time spent waiting is longer than {{clientTimeout}} (if it exists in the
parameters).
When the Client thread is interrupted, it ignores the exception:
{code:java}
// Check app status every 1 second.
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
LOG.debug("Thread sleep in monitoring loop interrupted");
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/357/ [Dec 15, 2020 4:36:11 AM] (Ayush Saxena) HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.(#2534) [Dec 15, 2020 4:42:23 AM] (Wei-Chiu Chuang) HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. [Dec 15, 2020 4:45:37 AM] (noreply) Hadoop-17413. Release elastic byte buffer pool at close [Dec 15, 2020 4:30:14 PM] (Szilard Nemeth) YARN-10526. RMAppManager CS Placement ignores parent path. Contributed by Gergely Pollak -1 overall The following subsystems voted -1: asflicense pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.hdfs.TestMultipleNNPortQOP hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor hadoop.hdfs.TestDecommission hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap hadoop.fs.contract.router.TestRouterHDFSContractMkdir hadoop.fs.contract.router.web.TestRouterWebHDFSContractMkdir hadoop.fs.contract.router.TestRouterHDFSContractRootDirectorySecure hadoop.hdfs.server.federation.router.TestRouterMultiRack hadoop.fs.contract.router.TestRouterHDFSContractGetFileStatus hadoop.fs.contract.router.TestRouterHDFSContractSetTimesSecure hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend hadoop.hdfs.server.federation.router.TestRouterFaultTolerant hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination hadoop.fs.contract.router.web.TestRouterWebHDFSContractConcat hadoop.fs.contract.router.TestRouterHDFSContractSeek hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate hadoop.hdfs.server.federation.router.TestRouterAllResolver hadoop.fs.contract.router.TestRouterHDFSContractAppendSecure hadoop.fs.contract.router.TestRouterHDFSContractDeleteSecure hadoop.fs.contract.router.TestRouterHDFSContractConcatSecure hadoop.fs.contract.router.web.TestRouterWebHDFSContractRename hadoop.fs.contract.router.TestRouterHDFSContractMkdirSecure hadoop.fs.contract.router.TestRouterHDFSContractOpenSecure hadoop.fs.contract.router.TestRouterHDFSContractCreateSecure hadoop.fs.contract.router.TestRouterHDFSContractConcat hadoop.hdfs.server.federation.router.TestRouterRpc hadoop.fs.contract.router.TestRouterHDFSContractCreate hadoop.fs.contract.router.TestRouterHDFSContractGetFileStatusSecure hadoop.fs.contract.router.web.TestRouterWebHDFSContractDelete hadoop.fs.contract.router.TestRouterHDFSContractRename hadoop.fs.contract.router.web.TestRouterWebHDFSContractOpen hadoop.hdfs.server.federation.router.TestRouterMountTableCacheRefresh hadoop.fs.contract.router.TestRouterHDFSContractOpen hadoop.fs.contract.router.TestRouterHDFSContractRenameSecure hadoop.fs.contract.router.TestRouterHDFSContractSetTimes hadoop.hdfs.server.federation.router.TestSafeMode hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.fs.contract.router.TestRouterHDFSContractSeekSecure hadoop.fs.contract.router.web.TestRouterWebHDFSContractSeek hadoop.fs.contract.router.TestRouterHDFSContractDelete hadoop.fs.contract.router.TestRouterHDFSContractAppend hadoop.fs.contract.router.web.TestRouterWebHDFSContractRootDirectory hadoop.yarn.server.webproxy.TestWebAppProxyServlet hadoop.yarn.server.webproxy.TestWebAppProxyServer hadoop.yarn.server.resourcemanager.TestSignalContainer hadoop.yarn.server.resourcemanager.rmapp.attempt.TestAMLivelinessMonitor
Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4
+1(binding) * checked hadoop-3.2.2-src.tar.gz * verified signature and checksums. * built on CentOS 8 and OpenJDK 8 by `mvn install -DskipTests -Pnative -Pdist`. * launched 3-node docker cluster with NN-HA and RM-HA enabled. * ran example MR jobs. * checked hadoop-3.2.2-site.tar.gz * verified signature and checksums. * skimmed the contents. While I think it is not critical, the top page of site documentation seemed to be rolled back from the one for 3.2.1: https://hadoop.apache.org/docs/r3.2.1/ . # I could not find the index.md.vm for 3.2.1 in branch-3.2.1 and rel/release-3.2.1. The step #4 of "Branching" section in HowToRelease is relevant instruction. https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease | 4. Release branch (branch-X.Y.Z) updates: |a. Update hadoop-project/src/site/markdown/index.md.vm to reflect the right versions, new features and big improvements. The instruction might be a bit unclear and tend to be missed by release manager. # I missed this on 2.10.1 release.. Thanks, Masatake Iwasaki On 2020/12/10 2:00, Xiaoqiao He wrote: Hi folks, The release candidate (RC4) for Hadoop-3.2.2 is available now. There are 10 commits[1] differences between RC4 and RC3[2]. The RC4 is available at: http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC4 The RC4 tag in github is here: https://github.com/apache/hadoop/tree/release-3.2.2-RC4 The maven artifacts are staged at: https://repository.apache.org/content/repositories/orgapachehadoop-1296 You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or https://people.apache.org/keys/committer/hexiaoqiao.asc directly. Please try the release and vote. I have done a simple testing with my pseudo cluster. * Verify gpg sign and md5sum. * Setup pseudo cluster with HDFS and YARN. * Run simple FsShell - mkdir/put/get/rename/mv. * Submit example mr job and check the result - Pi/wordcount. * Check the web UI of NameNode/DataNode/Resourcemanager/NodeManager. My +1 to start. Thanks, He Xiaoqiao [1] https://github.com/apache/hadoop/compare/release-3.2.2-RC3...release-3.2.2-RC4 [2] https://lists.apache.org/thread.html/rfb74c3a5d4f223c5804d8ee622829263740cd8701c8f3fc8b6f970af%40%3Chdfs-dev.hadoop.apache.org%3E [3] https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948 - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/ [Dec 15, 2020 10:08:20 PM] (Wei-Chiu Chuang) HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. -1 overall The following subsystems voted -1: asflicense hadolint jshint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml hadoop-build-tools/src/main/resources/checkstyle/suppressions.xml hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-tools/hadoop-azure/src/config/checkstyle.xml hadoop-tools/hadoop-resourceestimator/src/config/checkstyle.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml Failed junit tests : hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.namenode.ha.TestHAAppend hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.hdfs.TestMultipleNNPortQOP hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapred.TestMiniMRClasspath hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers hadoop.mapred.TestMRIntermediateDataEncryption hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver jshint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/diff-patch-jshint.txt [208K] cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/diff-compile-javac-root.txt [456K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/diff-patch-pylint.txt [60K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/diff-patch-shellcheck.txt [56K] shelldocs: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/whitespace-tabs.txt [1.3M] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/xml.txt [4.0K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/diff-javadoc-javadoc-root.txt [20K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [280K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/148/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K]
