Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/ [Aug 9, 2016 7:42:29 AM] (Arun Suresh) YARN-5457. Refactor DistributedScheduling framework to pull out common [Aug 9, 2016 10:34:09 AM] (vvasudev) YARN-5394. Remove bind-mount /etc/passwd for Docker containers. [Aug 9, 2016 3:55:35 PM] (jlowe) MAPREDUCE-6741. Add MR support to redact job conf properties. [Aug 9, 2016 4:56:21 PM] (kihwal) HDFS-10342. BlockManager#createLocatedBlocks should not check corrupt [Aug 9, 2016 4:58:54 PM] (rohithsharmaks) YARN-5474. Typo mistake in AMRMClient#getRegisteredTimeineClient API. [Aug 9, 2016 6:05:44 PM] (varunsaxena) MAPREDUCE-6750. Fix TestHSAdminServer#testRefreshSuperUserGroups (Kihwal [Aug 9, 2016 7:33:15 PM] (kihwal) HADOOP-13473. Tracing in IPC Server is broken. Contributed by Daryn [Aug 9, 2016 7:54:11 PM] (kihwal) HDFS-10738. Fix [Aug 9, 2016 8:42:32 PM] (kasha) HADOOP-13299. JMXJsonServlet is vulnerable to TRACE. (Haibo Chen via [Aug 9, 2016 10:51:19 PM] (lei) HDFS-10457. DataNode should not auto-format block pool directory if [Aug 9, 2016 10:59:22 PM] (lei) HDFS-10681. DiskBalancer: query command should report Plan file path [Aug 9, 2016 11:51:03 PM] (kasha) YARN-5343. TestContinuousScheduling#testSortedNodes fails [Aug 10, 2016 1:11:34 AM] (weichiu) HDFS-8224. Schedule a block for scanning if its metadata file is -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestRollingUpgrade hadoop.yarn.logaggregation.TestAggregatedLogFormat hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.api.impl.TestYarnClient Timed out junit tests : org.apache.hadoop.http.TestHttpServerLifecycle cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/diff-compile-javac-root.txt [172K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [120K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [148K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [24K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [36K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/129/artifact/
Re: [Release thread] 2.6.5 release activities
Thanks for organizing this, Chris! I don't believe HADOOP-13362 is needed since it's related to ContainerMetrics. ContainerMetrics weren't added until 2.7 by YARN-2984. YARN-4794 looks applicable to 2.6. The change drops right in except it has JDK7-isms (multi-catch clause), so it needs a slight change. Jason From: Chris Trezzo To: "common-...@hadoop.apache.org" ; hdfs-...@hadoop.apache.org; "mapreduce-dev@hadoop.apache.org" ; "yarn-...@hadoop.apache.org" Sent: Tuesday, August 9, 2016 7:32 PM Subject: [Release thread] 2.6.5 release activities Based on the sentiment in the "[DISCUSS] 2.6.x line releases" thread, I have moved forward with some of the initial effort in creating a 2.6.5 release. I am forking this thread so we have a dedicated 2.6.5 release thread. I have gone through the git logs and gathered a list of JIRAs that are in branch-2.7 but are missing from branch-2.6. I limited the diff to issues with a commit date after 1/26/2016. I did this because 2.6.4 was cut from branch-2.6 around that date (http://markmail.org/message/xmy7ebs6l3643o5e) and presumably issues that were committed to branch-2.7 before then were already looked at as part of 2.6.4. I have collected these issues in a spreadsheet and have given them an initial triage on whether they are candidates for a backport to 2.6.5. The spreadsheet is sorted by the status of the issues with the potential backport candidates at the top. Here is a link to the spreadsheet: https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC8hTRUemHvYPPzY/edit?usp=sharing As of now, I have identified 16 potential backport candidates. Please take a look at the list and let me know if there are any that you think should not be on the list, or ones that you think I have missed. This was just an initial high-level triage, so there could definitely be issues that are miss-labeled. As a side note: we still need to look at the pre-commit build for 2.6 and follow up with an addendum for HADOOP-12800. Thanks everyone! Chris Trezzo
Re: [Release thread] 2.6.5 release activities
Thanks Chris for bring up this discussion. Before we going to detail discussion of releasing 2.6.5, I have a quick question here: do we think it is necessary to continue to release branch-2.6, like 2.6.5, etc after 2.7 is out for more than 1 year. Any reason to not suggest users to upgrade to 2.7.3 releases for latest fixes which is in releasing now? My major concern on more release efforts on legacy branches is the same with my comments on other release plan before - it seems too many releases trains get planned at the same time window (2.6.x, 2.7.x, 2.8, 3.0-alpha, 3.1-beta, etc.). Not only user could get confusing on this, but also I suspect we don't have so many bandwidth in community to push forward so these releases in high quality during the same time window - just like Chris Douglas mentioned in another email thread on committer activity and bandwidth. IMO, may be it is better to focus on limited number of releases and move them faster? BTW, I agree with Jason that HADOOP-13362 is not needed for branch-2.6 unless we backport container metrics related patches there. Thanks, Junping From: Jason Lowe Sent: Wednesday, August 10, 2016 4:14 PM To: Chris Trezzo; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: Re: [Release thread] 2.6.5 release activities Thanks for organizing this, Chris! I don't believe HADOOP-13362 is needed since it's related to ContainerMetrics. ContainerMetrics weren't added until 2.7 by YARN-2984. YARN-4794 looks applicable to 2.6. The change drops right in except it has JDK7-isms (multi-catch clause), so it needs a slight change. Jason From: Chris Trezzo To: "common-...@hadoop.apache.org" ; hdfs-...@hadoop.apache.org; "mapreduce-dev@hadoop.apache.org" ; "yarn-...@hadoop.apache.org" Sent: Tuesday, August 9, 2016 7:32 PM Subject: [Release thread] 2.6.5 release activities Based on the sentiment in the "[DISCUSS] 2.6.x line releases" thread, I have moved forward with some of the initial effort in creating a 2.6.5 release. I am forking this thread so we have a dedicated 2.6.5 release thread. I have gone through the git logs and gathered a list of JIRAs that are in branch-2.7 but are missing from branch-2.6. I limited the diff to issues with a commit date after 1/26/2016. I did this because 2.6.4 was cut from branch-2.6 around that date (http://markmail.org/message/xmy7ebs6l3643o5e) and presumably issues that were committed to branch-2.7 before then were already looked at as part of 2.6.4. I have collected these issues in a spreadsheet and have given them an initial triage on whether they are candidates for a backport to 2.6.5. The spreadsheet is sorted by the status of the issues with the potential backport candidates at the top. Here is a link to the spreadsheet: https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC8hTRUemHvYPPzY/edit?usp=sharing As of now, I have identified 16 potential backport candidates. Please take a look at the list and let me know if there are any that you think should not be on the list, or ones that you think I have missed. This was just an initial high-level triage, so there could definitely be issues that are miss-labeled. As a side note: we still need to look at the pre-commit build for 2.6 and follow up with an addendum for HADOOP-12800. Thanks everyone! Chris Trezzo - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [Release thread] 2.6.5 release activities
Thanks Jason and Junping for the comments! I will update the spreadsheet for HADOOP-13362 and YARN-4794. As for continuing 2.6.x releases, please see the discussion in the "[DISCUSS] 2.6.x line releases" thread. Sean, Akira and Zhe all expressed interest in additional 2.6.x releases. I started this thread based off of that interest. I understand there is a burden to maintaining a large number of branches. I am not sure what the community's end-of-life policy is, but maybe we can issue a warning with the 2.6.5 release stating when we will stop maintaining the release line. This at least gives users some time to make migration plans to a newer version. On Wed, Aug 10, 2016 at 9:36 AM, Junping Du wrote: > Thanks Chris for bring up this discussion. > Before we going to detail discussion of releasing 2.6.5, I have a quick > question here: do we think it is necessary to continue to release > branch-2.6, like 2.6.5, etc after 2.7 is out for more than 1 year. Any > reason to not suggest users to upgrade to 2.7.3 releases for latest fixes > which is in releasing now? > My major concern on more release efforts on legacy branches is the same > with my comments on other release plan before - it seems too many releases > trains get planned at the same time window (2.6.x, 2.7.x, 2.8, 3.0-alpha, > 3.1-beta, etc.). Not only user could get confusing on this, but also I > suspect we don't have so many bandwidth in community to push forward so > these releases in high quality during the same time window - just like > Chris Douglas mentioned in another email thread on committer activity and > bandwidth. IMO, may be it is better to focus on limited number of releases > and move them faster? > > BTW, I agree with Jason that HADOOP-13362 is not needed for branch-2.6 > unless we backport container metrics related patches there. > > > Thanks, > > Junping > > From: Jason Lowe > Sent: Wednesday, August 10, 2016 4:14 PM > To: Chris Trezzo; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; > mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org > Subject: Re: [Release thread] 2.6.5 release activities > > Thanks for organizing this, Chris! > I don't believe HADOOP-13362 is needed since it's related to > ContainerMetrics. ContainerMetrics weren't added until 2.7 by YARN-2984. > YARN-4794 looks applicable to 2.6. The change drops right in except it > has JDK7-isms (multi-catch clause), so it needs a slight change. > > Jason > > From: Chris Trezzo > To: "common-...@hadoop.apache.org" ; > hdfs-...@hadoop.apache.org; "mapreduce-dev@hadoop.apache.org" < > mapreduce-dev@hadoop.apache.org>; "yarn-...@hadoop.apache.org" < > yarn-...@hadoop.apache.org> > Sent: Tuesday, August 9, 2016 7:32 PM > Subject: [Release thread] 2.6.5 release activities > > Based on the sentiment in the "[DISCUSS] 2.6.x line releases" thread, I > have moved forward with some of the initial effort in creating a 2.6.5 > release. I am forking this thread so we have a dedicated 2.6.5 release > thread. > > I have gone through the git logs and gathered a list of JIRAs that are in > branch-2.7 but are missing from branch-2.6. I limited the diff to issues > with a commit date after 1/26/2016. I did this because 2.6.4 was cut from > branch-2.6 around that date (http://markmail.org/message/xmy7ebs6l3643o5e) > and presumably issues that were committed to branch-2.7 before then were > already looked at as part of 2.6.4. > > I have collected these issues in a spreadsheet and have given them an > initial triage on whether they are candidates for a backport to 2.6.5. The > spreadsheet is sorted by the status of the issues with the potential > backport candidates at the top. Here is a link to the spreadsheet: > https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC > 8hTRUemHvYPPzY/edit?usp=sharing > > As of now, I have identified 16 potential backport candidates. Please take > a look at the list and let me know if there are any that you think should > not be on the list, or ones that you think I have missed. This was just an > initial high-level triage, so there could definitely be issues that are > miss-labeled. > > As a side note: we still need to look at the pre-commit build for 2.6 and > follow up with an addendum for HADOOP-12800. > > Thanks everyone! > Chris Trezzo > >
[jira] [Created] (MAPREDUCE-6751) Print a log message, when splitting is not possible because of an unsplittable compression
Peter Vary created MAPREDUCE-6751: - Summary: Print a log message, when splitting is not possible because of an unsplittable compression Key: MAPREDUCE-6751 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6751 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client, mrv1, mrv2 Affects Versions: 2.6.4 Reporter: Peter Vary Priority: Minor There should be a message logged in case of the mapreduce will only spam one mapper since the source file is compressed with an unsplitable algorithm -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6752) Inconsistent logging content and logging level
Nemo Chen created MAPREDUCE-6752:
Summary: Inconsistent logging content and logging level
Key: MAPREDUCE-6752
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6752
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.7.2
Reporter: Nemo Chen
Similar to previous issue HADOOP-3029, in the file:
hadoop-rel-release-2.7.2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
line 253, the whole method is for debugging purpose, therefore the log:
{code:borderStyle=solid}
LOG.info("We got asked to run a debug speculation scan.");
{code}
should be set to debug level instead of info level to avoid redundant
information.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
