[VOTE] Release Apache Hadoop 2.8.0 (RC3)
Hi all, With fix of HDFS-11431 get in, I've created a new release candidate (RC3) for Apache Hadoop 2.8.0. This is the next minor release to follow up 2.7.0 which has been released for more than 1 year. It comprises 2,900+ fixes, improvements, and new features. Most of these commits are released for the first time in branch-2. More information about the 2.8.0 release plan can be found here: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release New RC is available at: http://home.apache.org/~junping_du/hadoop-2.8.0-RC3 The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009 The maven artifacts are available via repository.apache.org at: https://repository.apache.org/content/repositories/orgapachehadoop-1057 Please try the release and vote; the vote will run for the usual 5 days, ending on 03/22/2017 PDT time. Thanks, Junping
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/ [Mar 16, 2017 2:08:30 PM] (stevel) HDFS-11431. hadoop-hdfs-client JAR does not include [Mar 16, 2017 2:30:10 PM] (jlowe) YARN-4051. ContainerKillEvent lost when container is still recovering [Mar 16, 2017 3:54:59 PM] (kihwal) HDFS-10601. Improve log message to include hostname when the NameNode is [Mar 16, 2017 7:06:51 PM] (stevel) Revert "HDFS-11431. hadoop-hdfs-client JAR does not include [Mar 16, 2017 7:20:46 PM] (jitendra) HDFS-11533. reuseAddress option should be used for child channels in [Mar 16, 2017 10:07:38 PM] (wang) HDFS-10530. BlockManager reconstruction work scheduling should correctly [Mar 16, 2017 11:08:32 PM] (liuml07) HADOOP-14191. Duplicate hadoop-minikdc dependency in hadoop-common [Mar 17, 2017 1:13:43 AM] (arp) HDFS-10394. move declaration of okhttp version from hdfs-client to -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.namenode.ha.TestHAAppend hadoop.hdfs.TestReadStripedFileWithMissingBlocks hadoop.yarn.server.nodemanager.containermanager.TestContainerManager hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.TestMiniYarnClusterNodeUtilization cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-compile-javac-root.txt [180K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-checkstyle-root.txt [17M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-patch-shellcheck.txt [24K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [296K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [36K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [60K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/348/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/ [Mar 16, 2017 2:08:30 PM] (stevel) HDFS-11431. hadoop-hdfs-client JAR does not include [Mar 16, 2017 2:30:10 PM] (jlowe) YARN-4051. ContainerKillEvent lost when container is still recovering [Mar 16, 2017 3:54:59 PM] (kihwal) HDFS-10601. Improve log message to include hostname when the NameNode is [Mar 16, 2017 7:06:51 PM] (stevel) Revert "HDFS-11431. hadoop-hdfs-client JAR does not include [Mar 16, 2017 7:20:46 PM] (jitendra) HDFS-11533. reuseAddress option should be used for child channels in [Mar 16, 2017 10:07:38 PM] (wang) HDFS-10530. BlockManager reconstruction work scheduling should correctly [Mar 16, 2017 11:08:32 PM] (liuml07) HADOOP-14191. Duplicate hadoop-minikdc dependency in hadoop-common [Mar 17, 2017 1:13:43 AM] (arp) HDFS-10394. move declaration of okhttp version from hdfs-client to -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.security.TestRaceWhenRelogin hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.app.TestRuntimeEstimators hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.mapreduce.TestMRJobClient Timed out junit tests : org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.yarn.client.api.impl.TestOpportunisticContainerAllocation compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-compile-root.txt [132K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-compile-root.txt [132K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-compile-root.txt [132K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [136K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [236K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [16K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [72K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [324K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell
Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
Hi Junping, Thank you for working on this. +1 (Non-Binding) I verified the following: 1. Deployed on a 3 node cluster with 2 node managers. 2. Configured linux container executor 3. Configured fair scheduler 4. Ran Pi job and verified the results 5. Ran multiple Yarn applications and verified the results Thank you, Miklos Szegedi On Fri, Mar 17, 2017 at 2:18 AM, Junping Du wrote: > Hi all, > With fix of HDFS-11431 get in, I've created a new release candidate > (RC3) for Apache Hadoop 2.8.0. > > This is the next minor release to follow up 2.7.0 which has been > released for more than 1 year. It comprises 2,900+ fixes, improvements, and > new features. Most of these commits are released for the first time in > branch-2. > > More information about the 2.8.0 release plan can be found here: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release > > New RC is available at: http://home.apache.org/~ > junping_du/hadoop-2.8.0-RC3 > > The RC tag in git is: release-2.8.0-RC3, and the latest commit id > is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009 > > The maven artifacts are available via repository.apache.org at: > https://repository.apache.org/content/repositories/orgapachehadoop-1057 > > Please try the release and vote; the vote will run for the usual 5 > days, ending on 03/22/2017 PDT time. > > Thanks, > > Junping >
[jira] [Created] (MAPREDUCE-6864) Hadoop streaming creates 2 mappers when the input has only one block
Daniel Templeton created MAPREDUCE-6864: --- Summary: Hadoop streaming creates 2 mappers when the input has only one block Key: MAPREDUCE-6864 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6864 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.7.3 Reporter: Daniel Templeton If a streaming job is run against input that is less than 2 blocks, 2 mappers will be created, both operating on the same split, both producing (duplicate) output. In some cases the second mapper will consistently fail. I've not seen the failure with input less than 10 bytes or more than a couple MB. I have seen it with a 4kB input. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
+1 Thanks, Junping, for your efforts to get this release out. I downloaded and built the source, and i did the following manual testing on a 2-node pseudo cluster: - Streaming job - Inter-queue (cross-queue) preemption--verified that only the expected amount of preemptions occured. - Inter-queue (in-queue) preemption with higher priority apps preempting lower ones. - Limited node label testing. - Yarn distributed shell, both with and without keeping containers across AM restart. - Killing apps from Application UI From: Junping Du To: "common-...@hadoop.apache.org" ; "hdfs-...@hadoop.apache.org" ; "yarn-...@hadoop.apache.org" ; "mapreduce-dev@hadoop.apache.org" Sent: Friday, March 17, 2017 4:18 AM Subject: [VOTE] Release Apache Hadoop 2.8.0 (RC3) Hi all, With fix of HDFS-11431 get in, I've created a new release candidate (RC3) for Apache Hadoop 2.8.0. This is the next minor release to follow up 2.7.0 which has been released for more than 1 year. It comprises 2,900+ fixes, improvements, and new features. Most of these commits are released for the first time in branch-2. More information about the 2.8.0 release plan can be found here: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release New RC is available at: http://home.apache.org/~junping_du/hadoop-2.8.0-RC3 The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009 The maven artifacts are available via repository.apache.org at: https://repository.apache.org/content/repositories/orgapachehadoop-1057 Please try the release and vote; the vote will run for the usual 5 days, ending on 03/22/2017 PDT time. Thanks, Junping - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
Thanks for the new RC, Junping. I built from source and tried it out on a 2-node cluster with HA enabled. I ran a pi job and some streaming jobs. I tested that localization and failover work correctly, and I played a little with the YARN and HDFS web UIs. I did encounter an old friend of mine, which is that if you submit a streaming job with input that is only 1 block, you will nonetheless get 2 mappers that both process the same split. What's new this time is that the second mapper was consistently failing on certain input sizes. I (re)verified that the issue also exists is 2.7.3, so it's not a regression. I'm pretty sure it's been there since at least 2.6.0. I filed MAPREDUCE-6864 for it. Given that my issue was not a regression, I'm +1 on the RC. Daniel On 3/17/17 2:18 AM, Junping Du wrote: Hi all, With fix of HDFS-11431 get in, I've created a new release candidate (RC3) for Apache Hadoop 2.8.0. This is the next minor release to follow up 2.7.0 which has been released for more than 1 year. It comprises 2,900+ fixes, improvements, and new features. Most of these commits are released for the first time in branch-2. More information about the 2.8.0 release plan can be found here: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release New RC is available at: http://home.apache.org/~junping_du/hadoop-2.8.0-RC3 The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009 The maven artifacts are available via repository.apache.org at: https://repository.apache.org/content/repositories/orgapachehadoop-1057 Please try the release and vote; the vote will run for the usual 5 days, ending on 03/22/2017 PDT time. Thanks, Junping - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
Thanks Junping for doing this. +1 (non-binding) 0. Download the src tar.gz file; checked the MD5 checksum 1. Build Hadoop from source successfully 2. Deploy a single node cluster and start the cluster successfully 3. Operate the HDFS from command line: ls, put, distcp, dfsadmin etc 4. Run hadoop mapreduce examples: grep 5. Operate AWS S3 using S3A schema from commandline: ls, cat, distcp 6. Check the HDFS service logs L > On Mar 17, 2017, at 2:18 AM, Junping Du wrote: > > Hi all, > With fix of HDFS-11431 get in, I've created a new release candidate (RC3) > for Apache Hadoop 2.8.0. > > This is the next minor release to follow up 2.7.0 which has been released > for more than 1 year. It comprises 2,900+ fixes, improvements, and new > features. Most of these commits are released for the first time in branch-2. > > More information about the 2.8.0 release plan can be found here: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release > > New RC is available at: > http://home.apache.org/~junping_du/hadoop-2.8.0-RC3 > > The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: > 91f2b7a13d1e97be65db92ddabc627cc29ac0009 > > The maven artifacts are available via repository.apache.org at: > https://repository.apache.org/content/repositories/orgapachehadoop-1057 > > Please try the release and vote; the vote will run for the usual 5 days, > ending on 03/22/2017 PDT time. > > Thanks, > > Junping
Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
+1 (binding) - Verfied signatures and digests- Performed a native build from the release tag- Deployed to a single node cluster- Ran some sample jobs Jason On Friday, March 17, 2017 4:18 AM, Junping Du wrote: Hi all, With fix of HDFS-11431 get in, I've created a new release candidate (RC3) for Apache Hadoop 2.8.0. This is the next minor release to follow up 2.7.0 which has been released for more than 1 year. It comprises 2,900+ fixes, improvements, and new features. Most of these commits are released for the first time in branch-2. More information about the 2.8.0 release plan can be found here: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release New RC is available at: http://home.apache.org/~junping_du/hadoop-2.8.0-RC3 The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009 The maven artifacts are available via repository.apache.org at: https://repository.apache.org/content/repositories/orgapachehadoop-1057 Please try the release and vote; the vote will run for the usual 5 days, ending on 03/22/2017 PDT time. Thanks, Junping
