Branch-3.1.0 Created
Devs, Today I updated fix version of all 3.1.0 JIRAs and moved all unresolved JIRA with target version = 3.1.0 to 3.2.0. I've created branch-3.1.0 and working on the RC0 now (still got various of issues while doing the release, but hopefully, I can get the RC0 done soon). If you have *ANY fix* want to commit to branch-3.1.0, please let me know. branch-3.1 is open for bug fixes. Let me know if you have any questions/thoughts. Thanks, Wangda
RE: [VOTE] Release Apache Hadoop 3.0.1 (RC1)
Thanks Eddy for the effort! +1 (binding) - Downloaded binary tarball, verified checksums, verified signature - Started a pseudo distributed hdfs and YARN cluster on Mac - verified hadoop runtime version - Check webUI - Run Pi - Verified basic hdfs operations - Verified basic erasure coding command(on directory, not file) - Verified copy/download file between HDFS and Aliyun OSS Bests, Sammi Chen -Original Message- From: Gabor Bota [mailto:gabor.b...@cloudera.com] Sent: Wednesday, March 21, 2018 7:36 AM To: Yufei GuCc: Eric Payne ; Hadoop Common ; Hdfs-dev ; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; Lei Xu Subject: Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1) Thanks for working on this Eddy! +1 (non-binding) I've done the following: - checked out git tag release-3.0.1-RC1 - built from source on Mac OS X 10.13.3, openjdk 1.8.0_144 (zulu) - deployed on a 3 node cluster (HDFS HA, Non-HA YARN) - verified example pi job - ran teragen, terasort and teravalidate without errors Regards, Gabor Bota On Tue, Mar 20, 2018 at 2:07 PM, Yufei Gu wrote: > Thanks Eddy! > > +1 (non-binding) > >- Downloaded the hadoop-3.0.1.tar.gz from > http://home.apache.org/~lei/hadoop-3.0.1-RC1/ >- Started a pseudo cluster inside one docker container >- Verified distributed shell >- Verified mapreduce pi job >- Sanity check RM WebUI > > Best, > > Yufei > > On Tue, Mar 20, 2018 at 9:32 AM, Eric Payne invalid> > wrote: > > > Thanks for working on this release! > > +1 (binding) > > I tested the following: > > - yarn distributed shell job > > > > - yarn streaming job > > > > - inter-queue preemption > > > > - compared behavior of fair and fifo ordering policy > > > > - both userlimit_first mode and priority_first mode of intra-queue > > preemption > > > > Eric Payne > > > > > > > > On Saturday, March 17, 2018, 11:11:32 PM CDT, Lei Xu > > > > > wrote: > > > > Hi, all > > > > I've created release candidate RC-1 for Apache Hadoop 3.0.1 > > > > Apache Hadoop 3.0.1 will be the first bug fix release for Apache > > Hadoop 3.0 release. It includes 49 bug fixes and security fixes, > > which include 12 blockers and 17 are critical. > > > > Please note: > > * HDFS-12990. Change default NameNode RPC port back to 8020. It > > makes incompatible changes to Hadoop 3.0.0. After 3.0.1 releases, > > Apache Hadoop 3.0.0 will be deprecated due to this change. > > > > The release page is: > > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Releas > > e > > > > New RC is available at: > > http://home.apache.org/~lei/hadoop-3.0.1-RC1/ > > > > The git tag is release-3.0.1-RC1, and the latest commit is > > 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0 > > > > The maven artifacts are available at: > > https://repository.apache.org/content/repositories/orgapachehadoop-1 > > 081/ > > > > Please try the release and vote; the vote will run for the usual 5 > > days, ending on 3/22/2017 6pm PST time. > > > > Thanks! > > > > > > - To unsubscribe, e-mail: > > mapreduce-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: > > mapreduce-dev-h...@hadoop.apache.org > > > > >
Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64
For more details, see https://builds.apache.org/job/hadoop-trunk-win/412/ [Mar 19, 2018 6:04:27 PM] (wangda) YARN-8002. Support NOT_SELF and ALL namespace types for allocation tag. [Mar 19, 2018 9:59:58 PM] (jlowe) YARN-8027. Setting hostname of docker container breaks for --net=host in [Mar 19, 2018 11:05:55 PM] (cdouglas) HADOOP-14667. Flexible Visual Studio support. Contributed by Allen [Mar 20, 2018 12:19:30 AM] (inigoiri) HDFS-13296. GenericTestUtils generates paths with drive letter in [Mar 20, 2018 5:13:40 AM] (weiy) HDFS-13215. RBF: Move Router to its own module. Contributed by Wei Yan [Mar 20, 2018 11:36:21 AM] (haibochen) YARN-8053. Add hadoop-distcp in exclusion in hbase-server dependencies -1 overall The following subsystems voted -1: unit The following subsystems are considered long running: (runtime bigger than 1h 00m 00s) unit Specific tests: Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static Failed junit tests : hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec hadoop.fs.contract.rawlocal.TestRawlocalContractAppend hadoop.fs.TestFsShellCopy hadoop.fs.TestFsShellList hadoop.fs.TestLocalFileSystem hadoop.fs.TestRawLocalFileSystemContract hadoop.fs.TestSymlinkLocalFSFileContext hadoop.fs.TestTrash hadoop.http.TestHttpServer hadoop.http.TestHttpServerLogs hadoop.io.compress.TestCodec hadoop.io.nativeio.TestNativeIO hadoop.ipc.TestIPC hadoop.ipc.TestProtoBufRpc hadoop.ipc.TestSocketFactory hadoop.metrics2.impl.TestStatsDMetrics hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal hadoop.metrics2.source.TestJvmMetrics hadoop.security.TestSecurityUtil hadoop.security.TestShellBasedUnixGroupsMapping hadoop.security.token.TestDtUtilShell hadoop.util.TestNativeCodeLoader hadoop.fs.TestWebHdfsFileContextMainOperations hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy hadoop.hdfs.crypto.TestHdfsCryptoStreams hadoop.hdfs.qjournal.client.TestQuorumJournalManager hadoop.hdfs.qjournal.server.TestJournalNode hadoop.hdfs.qjournal.server.TestJournalNodeSync hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage hadoop.hdfs.server.datanode.TestBlockRecovery hadoop.hdfs.server.datanode.TestBlockScanner hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics hadoop.hdfs.server.datanode.TestDataNodeFaultInjector hadoop.hdfs.server.datanode.TestDataNodeMetrics hadoop.hdfs.server.datanode.TestDataNodeUUID hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.datanode.TestHSync hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC hadoop.hdfs.server.mover.TestStorageMover hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport hadoop.hdfs.server.namenode.TestAddBlock hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands hadoop.hdfs.server.namenode.TestCheckpoint hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)
Thanks Eddy! +1 (non-binding) - Downloaded the hadoop-3.0.1.tar.gz from http://home.apache.org/~lei/hadoop-3.0.1-RC1/ - Started a pseudo cluster inside one docker container - Verified distributed shell - Verified mapreduce pi job - Sanity check RM WebUI Best, Yufei On Tue, Mar 20, 2018 at 9:32 AM, Eric Paynewrote: > Thanks for working on this release! > +1 (binding) > I tested the following: > - yarn distributed shell job > > - yarn streaming job > > - inter-queue preemption > > - compared behavior of fair and fifo ordering policy > > - both userlimit_first mode and priority_first mode of intra-queue > preemption > > Eric Payne > > > > On Saturday, March 17, 2018, 11:11:32 PM CDT, Lei Xu > wrote: > > Hi, all > > I've created release candidate RC-1 for Apache Hadoop 3.0.1 > > Apache Hadoop 3.0.1 will be the first bug fix release for Apache > Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which > include 12 > blockers and 17 are critical. > > Please note: > * HDFS-12990. Change default NameNode RPC port back to 8020. It makes > incompatible changes to Hadoop 3.0.0. After 3.0.1 releases, Apache > Hadoop 3.0.0 will be deprecated due to this change. > > The release page is: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release > > New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/ > > The git tag is release-3.0.1-RC1, and the latest commit is > 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0 > > The maven artifacts are available at: > https://repository.apache.org/content/repositories/orgapachehadoop-1081/ > > Please try the release and vote; the vote will run for the usual 5 > days, ending on 3/22/2017 6pm PST time. > > Thanks! > > - > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > >
Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)
Thanks for working on this release! +1 (binding) I tested the following: - yarn distributed shell job - yarn streaming job - inter-queue preemption - compared behavior of fair and fifo ordering policy - both userlimit_first mode and priority_first mode of intra-queue preemption Eric Payne On Saturday, March 17, 2018, 11:11:32 PM CDT, Lei Xuwrote: Hi, all I've created release candidate RC-1 for Apache Hadoop 3.0.1 Apache Hadoop 3.0.1 will be the first bug fix release for Apache Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which include 12 blockers and 17 are critical. Please note: * HDFS-12990. Change default NameNode RPC port back to 8020. It makes incompatible changes to Hadoop 3.0.0. After 3.0.1 releases, Apache Hadoop 3.0.0 will be deprecated due to this change. The release page is: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/ The git tag is release-3.0.1-RC1, and the latest commit is 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0 The maven artifacts are available at: https://repository.apache.org/content/repositories/orgapachehadoop-1081/ Please try the release and vote; the vote will run for the usual 5 days, ending on 3/22/2017 6pm PST time. Thanks! - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/ [Mar 19, 2018 7:02:37 AM] (sammi.chen) HADOOP-15262. AliyunOSS: move files under a directory in parallel when [Mar 19, 2018 6:04:27 PM] (wangda) YARN-8002. Support NOT_SELF and ALL namespace types for allocation tag. [Mar 19, 2018 9:59:58 PM] (jlowe) YARN-8027. Setting hostname of docker container breaks for --net=host in [Mar 19, 2018 11:05:55 PM] (cdouglas) HADOOP-14667. Flexible Visual Studio support. Contributed by Allen [Mar 20, 2018 12:19:30 AM] (inigoiri) HDFS-13296. GenericTestUtils generates paths with drive letter in -1 overall The following subsystems voted -1: findbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api org.apache.hadoop.yarn.api.records.Resource.getResources() may expose internal representation by returning Resource.resources At Resource.java:by returning Resource.resources At Resource.java:[line 234] Failed junit tests : hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.webapp.TestContainerLogsPage cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/diff-compile-javac-root.txt [288K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/whitespace-eol.txt [9.2M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/whitespace-tabs.txt [288K] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/diff-javadoc-javadoc-root.txt [760K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [300K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [48K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/726/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [84K] Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
RE: [VOTE] Release Apache Hadoop 3.0.1 (RC1)
Thanks Eddy for driving 3.0.1 release. I'm testing the release and will vote later. One thing I noticed that the JIRA links on page https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release is linked to the page itself, not the individual JIRA. I'm not sure if it's the design. Thanks, Sammi -Original Message- From: Lei Xu [mailto:l...@apache.org] Sent: Sunday, March 18, 2018 12:11 PM To: Hadoop Common; Hdfs-dev ; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: [VOTE] Release Apache Hadoop 3.0.1 (RC1) Hi, all I've created release candidate RC-1 for Apache Hadoop 3.0.1 Apache Hadoop 3.0.1 will be the first bug fix release for Apache Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which include 12 blockers and 17 are critical. Please note: * HDFS-12990. Change default NameNode RPC port back to 8020. It makes incompatible changes to Hadoop 3.0.0. After 3.0.1 releases, Apache Hadoop 3.0.0 will be deprecated due to this change. The release page is: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/ The git tag is release-3.0.1-RC1, and the latest commit is 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0 The maven artifacts are available at: https://repository.apache.org/content/repositories/orgapachehadoop-1081/ Please try the release and vote; the vote will run for the usual 5 days, ending on 3/22/2017 6pm PST time. Thanks! - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: About reset branch-3.1 to trunk before release.
Hi Sammi, Please go ahead to cherry-pick it. Will work on RC tomorrow. Thanks, Wangda On Mon, Mar 19, 2018 at 6:23 PM, Chen, Sammiwrote: > Thanks Wangda for driving the 3.1 release. > > HADOOP-15262 targets 3.1 release too. I didn't commit to the branch-3.1 > previously since I noticed that the branch will be reset to trunk. > > If you have no other concern, I will check pick HADOOP-15262 to the new > 3.1 branch today. > > > Thanks, > Sammi > -Original Message- > From: Wangda Tan [mailto:wheele...@gmail.com] > Sent: Tuesday, March 20, 2018 2:38 AM > To: Akira Ajisaka > Cc: Vinod Kumar Vavilapalli ; > yarn-...@hadoop.apache.org; Hdfs-dev ; > common-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org > Subject: Re: About reset branch-3.1 to trunk before release. > > Done JIRA fix version update: > > Moved all JIRAs with fixVersion = 3.2.0 to 3.1.0 except following few > fixes (which committed after 49c747ab187d0650143205ba57ca19607ec4c6bd) > > YARN-8002. Support NOT_SELF and ALL namespace types for allocation tag. > (Weiwe > i Yang via wangda) > > HADOOP-15262. AliyunOSS: move files under a directory in parallel when > rename a directory. Contributed by Jinhu Wu. > > MAPREDUCE-7066. TestQueue fails on Java9 > > YARN-8028. Support authorizeUserAccessToQueue in RMWebServices. > Contributed by Wangda Tan. > > YARN-8040. [UI2] New YARN UI webapp does not respect current pathname > for REST api. Contributed by Sunil G. > > Thanks, > Wangda > > On Mon, Mar 19, 2018 at 11:12 AM, Wangda Tan wrote: > > > Thanks Akira for the additional vote, > > > > With help from Apache Infra Team (Daniel Takamori), we just reset > > branch-3.1 to trunk (SHA: 49c747ab187d0650143205ba57ca19607ec4c6bd). > > Will update JIRA fix version shortly. > > > > - Wangda > > > > On Sun, Mar 18, 2018 at 6:10 PM, Akira Ajisaka > > > > wrote: > > > >> +1 for resetting branch-3.1. > >> > >> Thanks, > >> Akira > >> > >> > >> On 2018/03/18 12:51, Wangda Tan wrote: > >> > >>> Thanks for sharing your thoughts. > >>> > >>> We have done build and single node cluster deploy / test for the > >>> latest trunk code (commit: > >>> 49c747ab187d0650143205ba57ca19607ec4c6bd). Since there are no > >>> objections, so I will go ahead to do the branch replace. > >>> > >>> Since we don't have force push permission to release branches. I > >>> just filed > >>> https://issues.apache.org/jira/browse/INFRA-16204 to get help from > >>> Apache infra team. > >>> > >>> Please hold any commits to branch-3.1, will keep this email thread > >>> posted. > >>> > >>> Best, > >>> Wangda > >>> > >>> On Wed, Mar 14, 2018 at 3:14 PM, Vinod Kumar Vavilapalli < > >>> vino...@apache.org > >>> > wrote: > > >>> > >>> I see one new feature: https://issues.apache.org/jira/browse/YARN-7626 > : > Allow regular expression matching in container-executor.cfg for > devices and named docker volumes mount. > > There are 21 sub-tasks. There are three feature-type JIRAs in those > - https://issues.apache.org/jira/browse/YARN-7972, > https://issues.apache.org/jira/browse/YARN-7891 and > https://issues.apache.org/jira/browse/YARN-5015. These should be > okay - not major disrupting features. > > Everything else is either a bug-fix or an improvement so we should > be good. > > From the list, it doesn't look like resetting will destabilize > 3.1, +1 for doing this. > > Thanks > +Vinod > > On Mar 14, 2018, at 1:54 PM, Wangda Tan wrote: > > > > Hi mapreduce/yarn/common/hdfs-devs, > > > > As of now, we have all blockers done for 3.1.0 release [1]. The > > release > > > is running behind schedule due to a few security-related issues. > Because of > this and since branch-3.1 is cut 5 weeks before on Feb 8, trunk 3.2 > is already diverging. There're 64 commits in trunk but not in > branch-3.1. > [2] > > > > > I took a quick scan of them, most of them are good fixes which we > > should > > > bring to 3.1.0 as well. And this can also reduce differences > between > 3.2.0 > and 3.1.0 release for less maintenance burden in the future. > > > > > Unless anyone objects, we will reset branch-3.1 to trunk in 1-2 > > days and > > > cut RC after that. > > > > > Thoughts? > > > > - Wangda > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in > > (Blocker, > > > Critical) AND resolution = Unresolved AND "Target Version/s" = > 3.1.0 ORDER BY priority DESC > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in > > (3.2.0) > > > AND fixVersion not in (3.1.0) > > > > >>> > > >