Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)
Thanks for driving the release, Junping! +1 (binding) - Verified signatures and digests - Successfully performed a native build from source - Successfully deployed a single-node cluster with the timeline server - Ran some sample jobs and examined the web UI and job logs Jason On Mon, Sep 10, 2018 at 7:00 AM, 俊平堵 wrote: > Hi all, > > I've created the first release candidate (RC0) for Apache > Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It > includes 33 important fixes and improvements. > > > The RC artifacts are available at: > http://home.apache.org/~junping_du/hadoop-2.8.5-RC0 > > > The RC tag in git is: release-2.8.5-RC0 > > > > The maven artifacts are available via repository.apache.org< > http://repository.apache.org> at: > > https://repository.apache.org/content/repositories/orgapachehadoop-1140 > > > Please try the release and vote; the vote will run for the usual 5 > working > days, ending on 9/15/2018 PST time. > > > Thanks, > > > Junping >
[jira] [Created] (MAPREDUCE-7138) ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler
Oleksandr Shevchenko created MAPREDUCE-7138: --- Summary: ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler Key: MAPREDUCE-7138 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7138 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Oleksandr Shevchenko Assignee: Oleksandr Shevchenko MRAppBenchmark#benchmark2 test fails with the following exception: {noformat} java.lang.ClassCastException: org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark$ThrottledMRApp$ThrottledContainerAllocator cannot be cast to org.apache.hadoop.mapreduce.v2.app.rm.RMHeartbeatHandler at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getRMHeartbeatHandler(MRAppMaster.java:718) at org.apache.hadoop.mapreduce.v2.app.MRApp.createCommitterEventHandler(MRApp.java:665) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:391) at org.apache.hadoop.mapreduce.v2.app.MRApp.serviceInit(MRApp.java:266) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:294) at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:279) at org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:69) at org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark2(MRAppBenchmark.java:268) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) {noformat} since ThrottledContainerAllocator doesn't implement RMHeartbeatHandler interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/ [Sep 9, 2018 5:07:30 AM] (msingh) HDDS-410. ozone scmcli list is not working properly. Contributed by [Sep 9, 2018 5:50:26 PM] (brahma) HDFS-13862. RBF: Router logs are not capturing few of the dfsrouteradmin [Sep 10, 2018 12:45:31 AM] (wangda) YARN-8698. [Submarine] Failed to reset Hadoop home environment when [Sep 10, 2018 3:40:51 AM] (vinayakumarb) HDFS-13806. EC: No error message for unsetting EC policy of the [Sep 10, 2018 3:52:59 AM] (vinayakumarb) HDFS-13895. EC: Fix Intermittent Failure in -1 overall The following subsystems voted -1: asflicense findbugs pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine Unread field:FSBasedSubmarineStorageImpl.java:[line 39] Found reliance on default encoding in org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component):in org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component): new java.io.FileWriter(File) At YarnServiceJobSubmitter.java:[line 195] org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters, TaskType, Component) may fail to clean up java.io.Writer on checked exception Obligation to clean up resource created at YarnServiceJobSubmitter.java:to clean up java.io.Writer on checked exception Obligation to clean up resource created at YarnServiceJobSubmitter.java:[line 195] is not discharged org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceUtils.getComponentArrayJson(String, int, String) concatenates strings using + in a loop At YarnServiceUtils.java:using + in a loop At YarnServiceUtils.java:[line 72] Failed CTEST tests : test_test_libhdfs_threaded_hdfs_static test_libhdfs_threaded_hdfspp_test_shim_static Failed junit tests : hadoop.hdfs.TestLeaseRecovery2 hadoop.hdfs.client.impl.TestBlockReaderLocal hadoop.hdfs.web.TestWebHdfsTimeouts cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-compile-javac-root.txt [304K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-checkstyle-root.txt [17M] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/whitespace-eol.txt [9.4M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-submarine-warnings.html [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-hdds_client.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-hdds_framework.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/892/artifact/out/branch-findbugs-hadoop-ozone_c
[jira] [Created] (MAPREDUCE-7137) MRAppBenchmark.benchmark1() fails with NullPointerException
Oleksandr Shevchenko created MAPREDUCE-7137: --- Summary: MRAppBenchmark.benchmark1() fails with NullPointerException Key: MAPREDUCE-7137 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7137 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Oleksandr Shevchenko MRAppBenchmark.benchmark1() fails with NullPointerException: 1. We do not set any queue for this test. As the result we got the following exception: {noformat} 2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator (RMCommunicator.java:register(177)) - Exception while registering java.lang.NullPointerException at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123) at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172) at org.apache.avro.util.Utf8.(Utf8.java:39) at org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301) at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285) at org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72) at org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194) {noformat} 2. We override createSchedulerProxy method and do not set application priority that was added later by MAPREDUCE-6515. We got the following error: {noformat} java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280) at java.lang.Thread.run(Thread.java:748) {noformat} In both cases, the job never will be run and the test stuck and will not be finished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[VOTE] Release Apache Hadoop 2.8.5 (RC0)
Hi all, I've created the first release candidate (RC0) for Apache Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It includes 33 important fixes and improvements. The RC artifacts are available at: http://home.apache.org/~junping_du/hadoop-2.8.5-RC0 The RC tag in git is: release-2.8.5-RC0 The maven artifacts are available via repository.apache.org< http://repository.apache.org> at: https://repository.apache.org/content/repositories/orgapachehadoop-1140 Please try the release and vote; the vote will run for the usual 5 working days, ending on 9/15/2018 PST time. Thanks, Junping
[jira] [Created] (MAPREDUCE-7136) TestMRAppMetrics should shutdown DefaultMetricsSystem after completion
Oleksandr Shevchenko created MAPREDUCE-7136: --- Summary: TestMRAppMetrics should shutdown DefaultMetricsSystem after completion Key: MAPREDUCE-7136 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7136 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Oleksandr Shevchenko Attachments: image-2018-09-10-14-39-57-992.png TestMRAppMetrics should invoke shutdown method in DefaultMetricsSystem after completion. Since it can lead to failing other tests. For example, TestRMContainerAllocator#testReportedAppProgress fails when run after TestMRAppMetrics#testNames with the following error: {noformat} org.apache.hadoop.metrics2.MetricsException: Metrics source MRAppMetrics already exists! {noformat} !image-2018-09-10-14-39-57-992.png! We do not catch this on the trunk since the test TestRMContainerAllocator#testUnsupportedMapContainerRequirement run first and "DefaultMetricsSystem.shutdown();" invokes after completion. But since JUnit does not guarantee the order of the tests we should fix it. Also, this is affected by previous versions which run testReportedAppProgress first (I faced it on 2.7.0 version). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7135) TestTaskAttemptContainerRequest should reset UserGroupInformation
Oleksandr Shevchenko created MAPREDUCE-7135: --- Summary: TestTaskAttemptContainerRequest should reset UserGroupInformation Key: MAPREDUCE-7135 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7135 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Oleksandr Shevchenko TestTaskAttemptContainerRequest should reset UserGroupInformation after end since this test cache UserGroupInformation and can be cause of other test fail. For example, all test in TestRMContainerAllocator will be failed if we run these tests after TestTaskAttemptContainerRequest (or we can change "UserGroupInformation.setLoginUser(null)" with UserGroupInformation.reset() in this class). Also, I think the WARNING message in TestTaskAttemptContainerRequest can be removed since we can reset UserGroupInformation after each test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org