Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0
Thanks Brahma Reddy Battula for your great work here. Stephen fixed lease leak in namenode, and it is ready now: https://issues.apache.org/jira/browse/HDFS-14498. I think this affects 3.3.0-RC0. Would you check this? Sorry for reporting it so late. On Mon, Jul 13, 2020 at 2:53 PM Takanobu Asanuma wrote: > +1(non-binding) >- verified checksums >- succeeded in building the package with OpenJDK 8 >- started HDFS cluster with kerberos with OpenJDK 11 >- verified Web UIs (NN, DN, Router) >- Ran some operations of Router-based Federation with Security >- Ran some operations of Erasure Coding > > Thanks, > Takanobu > > > From: Bilwa S T > Sent: Monday, July 13, 2020 2:23 > To: Surendra Singh Lilhore; hemanth boyina > Cc: Brahma Reddy Battula; mapreduce-dev; Hdfs-dev; Hadoop Common; yarn-dev > Subject: RE: [VOTE] Release Apache Hadoop 3.3.0 - RC0 > > +1(non-binding) > > 1. Deployed 3 node cluster > 2. Browsed through Web UI (RM, NM) > 3. Executed Jobs (pi, wordcount, TeraGen, TeraSort) > 4. Verified basic yarn commands > > Thanks, > Bilwa > > -Original Message- > From: Surendra Singh Lilhore [mailto:surendralilh...@gmail.com] > Sent: 12 July 2020 18:32 > To: hemanth boyina > Cc: Iñigo Goiri ; Vinayakumar B < > vinayakum...@apache.org>; Brahma Reddy Battula ; > mapreduce-dev ; Hdfs-dev < > hdfs-...@hadoop.apache.org>; Hadoop Common ; > yarn-dev > Subject: Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0 > > +1(binding) > > Deployed HDFS and Yarn Cluster > > Verified basic shell commands > > Ran some jobs > > Verified UI > > -Surendra > > On Sat, Jul 11, 2020 at 9:41 PM hemanth boyina > > wrote: > > > +1(non-binding) > > Deployed Cluster with Namenodes and Router *)verified shell commands > > *)Executed various jobs *)Browsed UI's > > > > > > Thanks, > > HemanthBoyina > > > > > > On Sat, 11 Jul 2020, 00:05 Iñigo Goiri, wrote: > > > > > +1 (Binding) > > > > > > Deployed a cluster on Azure VMs with: > > > * 3 VMs with HDFS Namenodes and Routers > > > * 2 VMs with YARN Resource Managers > > > * 5 VMs with HDFS Datanodes and Node Managers > > > > > > Tests: > > > * Executed Tergagen+Terasort+Teravalidate. > > > * Executed wordcount. > > > * Browsed through the Web UI. > > > > > > > > > > > > On Fri, Jul 10, 2020 at 1:06 AM Vinayakumar B > > > > > > wrote: > > > > > > > +1 (Binding) > > > > > > > > -Verified all checksums and Signatures. > > > > -Verified site, Release notes and Change logs > > > > + May be changelog and release notes could be grouped based on > > > > the project at second level for better look (this needs to be > > > > supported > > from > > > > yetus) > > > > -Tested in x86 local 3-node docker cluster. > > > > + Built from source with OpenJdk 8 and Ubuntu 18.04 > > > > + Deployed 3 node docker cluster > > > > + Ran various Jobs (wordcount, Terasort, Pi, etc) > > > > > > > > No Issues reported. > > > > > > > > -Vinay > > > > > > > > On Fri, Jul 10, 2020 at 1:19 PM Sheng Liu > > > wrote: > > > > > > > > > +1 (non-binding) > > > > > > > > > > - checkout the "3.3.0-aarch64-RC0" binaries packages > > > > > > > > > > - started a clusters with 3 nodes VMs of Ubuntu 18.04 > > > > > ARM/aarch64, openjdk-11-jdk > > > > > > > > > > - checked some web UIs (NN, DN, RM, NM) > > > > > > > > > > - Executed a wordcount, TeraGen, TeraSort and TeraValidate > > > > > > > > > > - Executed a TestDFSIO job > > > > > > > > > > - Executed a Pi job > > > > > > > > > > BR, > > > > > Liusheng > > > > > > > > > > Zhenyu Zheng 于2020年7月10日周五 下午3:45写道： > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > - Verified all hashes and checksums > > > > > > - Tested on ARM platform for the following actions: > > > > > > + Built from source on Ubuntu 18.04, OpenJDK 8 > > > > > > + Deployed a pseudo cluster > > > > > > + Ran some example jobs(grep, wordcount, pi) > > > > > > + Ran teragen/terasort/teravalidate > > > > > > + Ran TestDFSIO job > > > > > > > > > > > > BR, > > > > > > > > > > > > Zhenyu > > > > > > > > > > > > On Fri, Jul 10, 2020 at 2:40 PM Akira Ajisaka > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > - Verified checksums and signatures. > > > > > > > - Built from the source with CentOS 7 and OpenJDK 8. > > > > > > > - Successfully upgraded HDFS to 3.3.0-RC0 in our development > > > cluster > > > > > > (with > > > > > > > RBF, security, and OpenJDK 11) for end-users. No issues > reported. > > > > > > > - The document looks good. > > > > > > > - Deployed pseudo cluster and ran some MapReduce jobs. > > > > > > > > > > > > > > Thanks, > > > > > > > Akira > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 7, 2020 at 7:27 AM Brahma Reddy Battula < > > > > bra...@apache.org > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi folks, > > > > > > > > > > > > > > > > This is the first release candidate for the first release > > > > > > > > of > > > Apache > > > >
[jira] [Created] (YARN-10350) TestUserGroupMappingPlacementRule fails
Akira Ajisaka created YARN-10350:
Summary: TestUserGroupMappingPlacementRule fails
Key: YARN-10350
URL: https://issues.apache.org/jira/browse/YARN-10350
Project: Hadoop YARN
Issue Type: Bug
Reporter: Akira Ajisaka
TestUserGroupMappingPlacementRule fails on trunk:
{noformat}
[INFO] Running
org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule
[ERROR] Tests run: 31, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 2.662
s <<< FAILURE! - in
org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule
[ERROR]
testResolvedQueueIsNotManaged(org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule)
Time elapsed: 0.03 s <<< ERROR!
java.lang.Exception: Unexpected exception,
expected but
was
at
org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:28)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.AssertionError: Queue expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at
org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule.verifyQueueMapping(TestUserGroupMappingPlacementRule.java:236)
at
org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule.testResolvedQueueIsNotManaged(TestUserGroupMappingPlacementRule.java:516)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
... 18 more
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0
Hi Xiaoqiao, Thanks for bringing this to my attention. It's too rare to occur this scenario and a couple of releases went after HDFS-4882(broken jira). So,You think it's fine to include in the next release? Or if you still insist we can go for another RC... On Mon, Jul 13, 2020 at 12:29 PM Xiaoqiao He wrote: > Thanks Brahma Reddy Battula for your great work here. > Stephen fixed lease leak in namenode, and it is ready now: > https://issues.apache.org/jira/browse/HDFS-14498. > I think this affects 3.3.0-RC0. Would you check this? > Sorry for reporting it so late. > > On Mon, Jul 13, 2020 at 2:53 PM Takanobu Asanuma > wrote: > >> +1(non-binding) >>- verified checksums >>- succeeded in building the package with OpenJDK 8 >>- started HDFS cluster with kerberos with OpenJDK 11 >>- verified Web UIs (NN, DN, Router) >>- Ran some operations of Router-based Federation with Security >>- Ran some operations of Erasure Coding >> >> Thanks, >> Takanobu >> >> >> From: Bilwa S T >> Sent: Monday, July 13, 2020 2:23 >> To: Surendra Singh Lilhore; hemanth boyina >> Cc: Brahma Reddy Battula; mapreduce-dev; Hdfs-dev; Hadoop Common; yarn-dev >> Subject: RE: [VOTE] Release Apache Hadoop 3.3.0 - RC0 >> >> +1(non-binding) >> >> 1. Deployed 3 node cluster >> 2. Browsed through Web UI (RM, NM) >> 3. Executed Jobs (pi, wordcount, TeraGen, TeraSort) >> 4. Verified basic yarn commands >> >> Thanks, >> Bilwa >> >> -Original Message- >> From: Surendra Singh Lilhore [mailto:surendralilh...@gmail.com] >> Sent: 12 July 2020 18:32 >> To: hemanth boyina >> Cc: Iñigo Goiri ; Vinayakumar B < >> vinayakum...@apache.org>; Brahma Reddy Battula ; >> mapreduce-dev ; Hdfs-dev < >> hdfs-...@hadoop.apache.org>; Hadoop Common ; >> yarn-dev >> Subject: Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0 >> >> +1(binding) >> >> Deployed HDFS and Yarn Cluster >> > Verified basic shell commands >> > Ran some jobs >> > Verified UI >> >> -Surendra >> >> On Sat, Jul 11, 2020 at 9:41 PM hemanth boyina < >> hemanthboyina...@gmail.com> >> wrote: >> >> > +1(non-binding) >> > Deployed Cluster with Namenodes and Router *)verified shell commands >> > *)Executed various jobs *)Browsed UI's >> > >> > >> > Thanks, >> > HemanthBoyina >> > >> > >> > On Sat, 11 Jul 2020, 00:05 Iñigo Goiri, wrote: >> > >> > > +1 (Binding) >> > > >> > > Deployed a cluster on Azure VMs with: >> > > * 3 VMs with HDFS Namenodes and Routers >> > > * 2 VMs with YARN Resource Managers >> > > * 5 VMs with HDFS Datanodes and Node Managers >> > > >> > > Tests: >> > > * Executed Tergagen+Terasort+Teravalidate. >> > > * Executed wordcount. >> > > * Browsed through the Web UI. >> > > >> > > >> > > >> > > On Fri, Jul 10, 2020 at 1:06 AM Vinayakumar B >> > > >> > > wrote: >> > > >> > > > +1 (Binding) >> > > > >> > > > -Verified all checksums and Signatures. >> > > > -Verified site, Release notes and Change logs >> > > > + May be changelog and release notes could be grouped based on >> > > > the project at second level for better look (this needs to be >> > > > supported >> > from >> > > > yetus) >> > > > -Tested in x86 local 3-node docker cluster. >> > > > + Built from source with OpenJdk 8 and Ubuntu 18.04 >> > > > + Deployed 3 node docker cluster >> > > > + Ran various Jobs (wordcount, Terasort, Pi, etc) >> > > > >> > > > No Issues reported. >> > > > >> > > > -Vinay >> > > > >> > > > On Fri, Jul 10, 2020 at 1:19 PM Sheng Liu >> > > wrote: >> > > > >> > > > > +1 (non-binding) >> > > > > >> > > > > - checkout the "3.3.0-aarch64-RC0" binaries packages >> > > > > >> > > > > - started a clusters with 3 nodes VMs of Ubuntu 18.04 >> > > > > ARM/aarch64, openjdk-11-jdk >> > > > > >> > > > > - checked some web UIs (NN, DN, RM, NM) >> > > > > >> > > > > - Executed a wordcount, TeraGen, TeraSort and TeraValidate >> > > > > >> > > > > - Executed a TestDFSIO job >> > > > > >> > > > > - Executed a Pi job >> > > > > >> > > > > BR, >> > > > > Liusheng >> > > > > >> > > > > Zhenyu Zheng 于2020年7月10日周五 下午3:45写道： >> > > > > >> > > > > > +1 (non-binding) >> > > > > > >> > > > > > - Verified all hashes and checksums >> > > > > > - Tested on ARM platform for the following actions: >> > > > > > + Built from source on Ubuntu 18.04, OpenJDK 8 >> > > > > > + Deployed a pseudo cluster >> > > > > > + Ran some example jobs(grep, wordcount, pi) >> > > > > > + Ran teragen/terasort/teravalidate >> > > > > > + Ran TestDFSIO job >> > > > > > >> > > > > > BR, >> > > > > > >> > > > > > Zhenyu >> > > > > > >> > > > > > On Fri, Jul 10, 2020 at 2:40 PM Akira Ajisaka >> > > > > > > > > >> > > > > wrote: >> > > > > > >> > > > > > > +1 (binding) >> > > > > > > >> > > > > > > - Verified checksums and signatures. >> > > > > > > - Built from the source with CentOS 7 and OpenJDK 8. >> > > > > > > - Successfully upgraded HDFS to 3.3.0-RC0 in our development >> > > cluster >> > > > > > (with >> > > > > > > RBF, security, and OpenJDK 11) for end-users. N
Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0
Thanks Brahma for putting this up, +1 (binding). * verified the signature and checksum of source tarball. * build from source tarball with native profile enabled on CentOS 7 and CentOS 8 with OpenJDK 8. * build documentation and skimmed the contents. * ran example jobs on pseudo-distributed cluster on CentOS 7 and CentOS 8. * ran example jobs on 3 nodes docker cluster with NN-HA and RM-HA enblaed on CentOS 7. Masatake Iwasaki On 2020/07/07 7:27, Brahma Reddy Battula wrote: Hi folks, This is the first release candidate for the first release of Apache Hadoop 3.3.0 line. It contains *1644[1]* fixed jira issues since 3.2.1 which include a lot of features and improvements(read the full set of release notes). Below feature additions are the highlights of the release. - ARM Support - Enhancements and new features on S3a,S3Guard,ABFS - Java 11 Runtime support and TLS 1.3. - Support Tencent Cloud COS File System. - Added security to HDFS Router. - Support non-volatile storage class memory(SCM) in HDFS cache directives - Support Interactive Docker Shell for running Containers. - Scheduling of opportunistic containers - A pluggable device plugin framework to ease vendor plugin development *The RC0 artifacts are at*: http://home.apache.org/~brahma/Hadoop-3.3.0-RC0/ *First release to include ARM binary, Have a check.* *RC tag is *release-3.3.0-RC0. *The maven artifacts are hosted here:* https://repository.apache.org/content/repositories/orgapachehadoop-1271/ *My public key is available here:* https://dist.apache.org/repos/dist/release/hadoop/common/KEYS The vote will run for 5 weekdays, until Tuesday, July 13 at 3:50 AM IST. I have done a few testing with my pseudo cluster. My +1 to start. Regards, Brahma Reddy Battula 1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.3.0) AND fixVersion not in (3.2.0, 3.2.1, 3.1.3) AND status = Resolved ORDER BY fixVersion ASC - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[VOTE] Release Apache Hadoop 3.1.4 (RC3)
Hi folks, I have put together a release candidate (RC3) for Hadoop 3.1.4. * The RC includes in addition to the previous ones: * fix of YARN-10347. Fix double locking in CapacityScheduler#reinitialize in branch-3.1 (https://issues.apache.org/jira/browse/YARN-10347) * the revert of HDFS-14941, as it caused HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. (https://issues.apache.org/jira/browse/HDFS-15421) * HDFS-15323, as requested. (https://issues.apache.org/jira/browse/HDFS-15323) * The RC is available at: http://people.apache.org/~gabota/hadoop-3.1.4-RC3/ The RC tag in git is here: https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC3 The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1274/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS and http://keys.gnupg.net/pks/lookup?op=get&search=0xB86249D83539B38C Please try the release and vote. The vote will run for 7 weekdays, until July 22. 2020. 23:00 CET. Thanks, Gabor - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0
Thanks Brahma, it makes sense for me to include that patch in the next release. +1 (non-binding). - Successfully built from source with native on CentOS6 with jdk1.8.112. - Setup a pseudo-distributed cluster with HDFS RBF arch and enable security using Kerberos on CentOS6. - Run simple HDFS Shell commands through Router(mkdir/chown/chmod/rename/put/get/ls/cat/stat/rmr). - Executed WordCount, Pi, Grep jobs. - Browsed the master WebUI (JN, NN, Router, RM). Thanks, He Xiaoqiao On Mon, Jul 13, 2020 at 5:52 PM Brahma Reddy Battula wrote: > Hi Xiaoqiao, > > Thanks for bringing this to my attention. > > It's too rare to occur this scenario and a couple of releases went after > HDFS-4882(broken jira). > > So,You think it's fine to include in the next release? Or if you still > insist we can go for another RC... > > > On Mon, Jul 13, 2020 at 12:29 PM Xiaoqiao He > wrote: > >> Thanks Brahma Reddy Battula for your great work here. >> Stephen fixed lease leak in namenode, and it is ready now: >> https://issues.apache.org/jira/browse/HDFS-14498. >> I think this affects 3.3.0-RC0. Would you check this? >> Sorry for reporting it so late. >> >> On Mon, Jul 13, 2020 at 2:53 PM Takanobu Asanuma >> wrote: >> >>> +1(non-binding) >>>- verified checksums >>>- succeeded in building the package with OpenJDK 8 >>>- started HDFS cluster with kerberos with OpenJDK 11 >>>- verified Web UIs (NN, DN, Router) >>>- Ran some operations of Router-based Federation with Security >>>- Ran some operations of Erasure Coding >>> >>> Thanks, >>> Takanobu >>> >>> >>> From: Bilwa S T >>> Sent: Monday, July 13, 2020 2:23 >>> To: Surendra Singh Lilhore; hemanth boyina >>> Cc: Brahma Reddy Battula; mapreduce-dev; Hdfs-dev; Hadoop Common; >>> yarn-dev >>> Subject: RE: [VOTE] Release Apache Hadoop 3.3.0 - RC0 >>> >>> +1(non-binding) >>> >>> 1. Deployed 3 node cluster >>> 2. Browsed through Web UI (RM, NM) >>> 3. Executed Jobs (pi, wordcount, TeraGen, TeraSort) >>> 4. Verified basic yarn commands >>> >>> Thanks, >>> Bilwa >>> >>> -Original Message- >>> From: Surendra Singh Lilhore [mailto:surendralilh...@gmail.com] >>> Sent: 12 July 2020 18:32 >>> To: hemanth boyina >>> Cc: Iñigo Goiri ; Vinayakumar B < >>> vinayakum...@apache.org>; Brahma Reddy Battula ; >>> mapreduce-dev ; Hdfs-dev < >>> hdfs-...@hadoop.apache.org>; Hadoop Common ; >>> yarn-dev >>> Subject: Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0 >>> >>> +1(binding) >>> >>> Deployed HDFS and Yarn Cluster >>> > Verified basic shell commands >>> > Ran some jobs >>> > Verified UI >>> >>> -Surendra >>> >>> On Sat, Jul 11, 2020 at 9:41 PM hemanth boyina < >>> hemanthboyina...@gmail.com> >>> wrote: >>> >>> > +1(non-binding) >>> > Deployed Cluster with Namenodes and Router *)verified shell commands >>> > *)Executed various jobs *)Browsed UI's >>> > >>> > >>> > Thanks, >>> > HemanthBoyina >>> > >>> > >>> > On Sat, 11 Jul 2020, 00:05 Iñigo Goiri, wrote: >>> > >>> > > +1 (Binding) >>> > > >>> > > Deployed a cluster on Azure VMs with: >>> > > * 3 VMs with HDFS Namenodes and Routers >>> > > * 2 VMs with YARN Resource Managers >>> > > * 5 VMs with HDFS Datanodes and Node Managers >>> > > >>> > > Tests: >>> > > * Executed Tergagen+Terasort+Teravalidate. >>> > > * Executed wordcount. >>> > > * Browsed through the Web UI. >>> > > >>> > > >>> > > >>> > > On Fri, Jul 10, 2020 at 1:06 AM Vinayakumar B >>> > > >>> > > wrote: >>> > > >>> > > > +1 (Binding) >>> > > > >>> > > > -Verified all checksums and Signatures. >>> > > > -Verified site, Release notes and Change logs >>> > > > + May be changelog and release notes could be grouped based on >>> > > > the project at second level for better look (this needs to be >>> > > > supported >>> > from >>> > > > yetus) >>> > > > -Tested in x86 local 3-node docker cluster. >>> > > > + Built from source with OpenJdk 8 and Ubuntu 18.04 >>> > > > + Deployed 3 node docker cluster >>> > > > + Ran various Jobs (wordcount, Terasort, Pi, etc) >>> > > > >>> > > > No Issues reported. >>> > > > >>> > > > -Vinay >>> > > > >>> > > > On Fri, Jul 10, 2020 at 1:19 PM Sheng Liu >>> > > wrote: >>> > > > >>> > > > > +1 (non-binding) >>> > > > > >>> > > > > - checkout the "3.3.0-aarch64-RC0" binaries packages >>> > > > > >>> > > > > - started a clusters with 3 nodes VMs of Ubuntu 18.04 >>> > > > > ARM/aarch64, openjdk-11-jdk >>> > > > > >>> > > > > - checked some web UIs (NN, DN, RM, NM) >>> > > > > >>> > > > > - Executed a wordcount, TeraGen, TeraSort and TeraValidate >>> > > > > >>> > > > > - Executed a TestDFSIO job >>> > > > > >>> > > > > - Executed a Pi job >>> > > > > >>> > > > > BR, >>> > > > > Liusheng >>> > > > > >>> > > > > Zhenyu Zheng 于2020年7月10日周五 下午3:45写道： >>> > > > > >>> > > > > > +1 (non-binding) >>> > > > > > >>> > > > > > - Verified all hashes and checksums >>> > > > > > - Tested on ARM platform for the following actions: >>> > > > > > + Built from source on Ubun
Apache Hadoop qbt Report: branch2.10+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/746/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint jshint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml findbugs : module:hadoop-yarn-project/hadoop-yarn Useless object stored in variable removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:[line 664] org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache() makes inefficient use of keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:[line 741] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus() makes inefficient use of keySet iterator instead of entrySet iterator At ContainerLocalizer.java:keySet iterator instead of entrySet iterator At ContainerLocalizer.java:[line 359] org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics is a mutable collection which should be package protected At ContainerMetrics.java:which should be package protected At ContainerMetrics.java:[line 134] Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] org.apache.hadoop.yarn.state.StateMachineFactory.generateStateGraph(String) makes inefficient use of keySet iterator instead of entrySet iterator At StateMachineFactory.java:keySet iterator instead of entrySet iterator At StateMachineFactory.java:[line 505] findbugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common org.apache.hadoop.yarn.state.StateMachineFactory.generateStateGraph(String) makes inefficient use of keySet iterator instead of entrySet iterator At StateMachineFactory.java:keySet iterator instead of entrySet iterator At StateMachineFactory.java:[line 505] findbugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Useless object stored in variable removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:[line 664] org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache() makes inefficient use of keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:[line 741] org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus() makes inefficient use of keySet iterator instead of entrySet iterator At ContainerLocalizer.java:keySet iterator instead of entrySet iterator At ContainerLocalizer.java:[line 359] org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics is a mutable collection which should be package protected At ContainerMetrics.java:which should be package protected At ContainerMetrics.java:[line 134] Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.
Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0
Thanks Brahma for getting this out! +1 (binding) Verified the following and looks fine to me. * Built from source with CentOS 7.4 and OpenJDK 1.8.0_232. * Deployed 3-node cluster. * Verified HDFS web UIs. * Tried out a few basic hdfs shell commands. * Ran sample Terasort, Wordcount MR jobs. -Rakesh- On Tue, Jul 7, 2020 at 3:57 AM Brahma Reddy Battula wrote: > Hi folks, > > This is the first release candidate for the first release of Apache > Hadoop 3.3.0 > line. > > It contains *1644[1]* fixed jira issues since 3.2.1 which include a lot of > features and improvements(read the full set of release notes). > > Below feature additions are the highlights of the release. > > - ARM Support > - Enhancements and new features on S3a,S3Guard,ABFS > - Java 11 Runtime support and TLS 1.3. > - Support Tencent Cloud COS File System. > - Added security to HDFS Router. > - Support non-volatile storage class memory(SCM) in HDFS cache directives > - Support Interactive Docker Shell for running Containers. > - Scheduling of opportunistic containers > - A pluggable device plugin framework to ease vendor plugin development > > *The RC0 artifacts are at*: > http://home.apache.org/~brahma/Hadoop-3.3.0-RC0/ > > *First release to include ARM binary, Have a check.* > *RC tag is *release-3.3.0-RC0. > > > *The maven artifacts are hosted here:* > https://repository.apache.org/content/repositories/orgapachehadoop-1271/ > > *My public key is available here:* > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > The vote will run for 5 weekdays, until Tuesday, July 13 at 3:50 AM IST. > > > I have done a few testing with my pseudo cluster. My +1 to start. > > > > Regards, > Brahma Reddy Battula > > > 1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.3.0) AND > fixVersion not in (3.2.0, 3.2.1, 3.1.3) AND status = Resolved ORDER BY > fixVersion ASC >
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/202/ [Jul 12, 2020 6:50:04 AM] (noreply) HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. [Jul 12, 2020 7:10:12 AM] (noreply) HDFS-15447 RBF: Add top real owners metrics for delegation tokens (#2110) -1 overall The following subsystems voted -1: asflicense findbugs pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml findbugs : module:hadoop-yarn-project/hadoop-yarn Uncallable method org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage$1.getInstance() defined in anonymous class At TestTimelineReaderWebServicesHBaseStorage.java:anonymous class At TestTimelineReaderWebServicesHBaseStorage.java:[line 87] Dead store to entities in org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl) At TestTimelineReaderHBaseDown.java:org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl) At TestTimelineReaderHBaseDown.java:[line 190] findbugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Uncallable method org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage$1.getInstance() defined in anonymous class At TestTimelineReaderWebServicesHBaseStorage.java:anonymous class At TestTimelineReaderWebServicesHBaseStorage.java:[line 87] Dead store to entities in org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl) At TestTimelineReaderHBaseDown.java:org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl) At TestTimelineReaderHBaseDown.java:[line 190] findbugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests Uncallable method org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage$1.getInstance() defined in anonymous class At TestTimelineReaderWebServicesHBaseStorage.java:anonymous class At TestTimelineReaderWebServicesHBaseStorage.java:[line 87] Dead store to entities in org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl) At TestTimelineReaderHBaseDown.java:org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl) At TestTimelineReaderHBaseDown.java:[line 190] findbugs : module:hadoop-yarn-project Uncallable method org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage$1.getInstance() defined in anonymous class At TestTimelineReaderWebServicesHBaseStorage.java:anonymous class At TestTimelineReaderWebServicesHBaseStorage.java:[line 87] Dead store to entities in org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl) At TestTimelineReaderHBaseDown.java:org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown.checkQuery(HBaseTimelineReaderImpl) At TestTimelineReaderHBaseDown.java:[line 190] findbugs : module:hadoop-cloud-storage-project/hadoop-cos org.apache.hadoop.fs.cosn.CosNInputStream$ReadBuffer.getBuffer() may expose internal representation by returning CosNInputStream$ReadBuffer.buffer At CosNInputStream.java:by returning CosNInputStream$ReadBuffer.buffer At CosNInputStream.java:[line 87] findbugs : module:hadoop-cloud-storage-p
Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0
+1 (binding) - Built from source on RHEL 7.6 - Deployed on a single-node cluster - Verified DefaultContainerRuntime - Verified RuncContainerRuntime (after setting things up with the docker-to-squash tool available on YARN-9564) Eric
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/203/ [Jul 13, 2020 6:12:48 AM] (Xiaoqiao He) HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. [Jul 13, 2020 12:30:02 PM] (Steve Loughran) HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext. [Jul 13, 2020 6:07:48 PM] (noreply) HADOOP-17105. S3AFS - Do not attempt to resolve symlinks in globStatus (#2113) [Jul 13, 2020 6:57:50 PM] (ericp) YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) [Jul 13, 2020 7:55:34 PM] (Hanisha Koneru) HADOOP-17116. Skip Retry INFO logging on first failover from a proxy [Jul 13, 2020 11:10:39 PM] (Eric Badger) YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by [Error replacing 'FILE' - Workspace is not accessible] - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
