Re: [Release thread] 2.6.5 release activities
Thanks Chris! I'll help Chris to get those JIRAs marked in his spreadsheet committed. We'll cut the release branch shortly after that. If you have any critical change that should be made part of 2.6.5 (CVE patches included), please reach out to us and commit the changes. If all things go well, we'd like to cut the branch in a few days. Thanks, Sangjin On Fri, Sep 9, 2016 at 1:24 PM, Chris Trezzo wrote: > Hi all, > > I wanted to give an update on the Hadoop 2.6.5 release efforts. > > Here is what has been done so far: > > 1. I have gone through all of the potential backports and recorded the > commit hashes for each of them from the branch that seems the most > appropriate (i.e. if there was a backport to 2.7.x then I used the hash > from the backport). > > 2. I verified if the cherry pick for each commit is clean. This was best > effort as some of the patches are in parts of the code that I am less > familiar with. This is recorded in the public spread sheet here: > https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3ol > WpOCo6EBAey1WYC8hTRUemHvYPPzY/edit?usp=sharing > > I am going to need help from committers to get these backports committed. > If there are any committers that have some spare cycles, especially if you > were involved with the initial commit for one of these issues, please look > at the spreadsheet and volunteer to backport one of the issues. > > As always, please let me know if you have any questions or feel that I have > missed something. > > Thank you! > Chris Trezzo > > On Mon, Aug 15, 2016 at 10:55 AM, Allen Wittenauer < > a...@effectivemachines.com > > wrote: > > > > > > On Aug 12, 2016, at 8:19 AM, Junping Du wrote: > > > > > > In this community, we are so aggressive to drop Java 7 support in > 3.0.x > > release. Here, why we are so conservative to keep releasing new bits to > > support Java 6? > > > > I don't view a group of people putting bug fixes into a micro > > release as particularly conservative. If a group within the community > > wasn't interested in doing it, 2.6.5 wouldn't be happening. > > > > But let's put the releases into context, because I think it tells > > a more interesting story. > > > > * hadoop 2.6.x = EOLed JREs (6,7) > > * hadoop 2.7 -> hadoop 2.x = transitional (7,8) > > * hadoop 3.x = JRE 8 > > * hadoop 4.x = JRE 9 > > > > There are groups of people still using JDK6 and they want bug > > fixes in a maintenance release. Boom, there's 2.6.x. > > > > Hadoop 3.x has been pushed off for years for "reasons". So we > > still have releases coming off of branch-2. If 2.7 had been released as > > 3.x, this chart would look less weird. But it wasn't thus 2.x has this > > weird wart in the middle of that supports JDK7 and JDK8. Given the > public > > policy and roadmaps of at least one major vendor at the time of this > > writing, we should expect to see JDK7 support for at least the next two > > years after 3.x appears. Bang, there's 2.x, where x is some large number. > > > > Then there is the future. People using JRE 8 want to use newer > > dependencies. A reasonable request. Some of these dependency updates > won't > > work with JRE 7. We can't do that in hadoop 2.x in any sort of > compatible > > way without breaking the universe. (Tons of JIRAs on this point.) This > > means we can only do it in 3.x (re: Hadoop Compatibility Guidelines). > > Kapow, there's 3.x > > > > The log4j community has stated that v1 won't work with JDK9. In > > turn, this means we'll need to upgrade to v2 at some point. Upgrading to > > v2 will break the log4j properties file (and maybe other things?). > Another > > incompatible change and it likely won't appear until Apache Hadoop v4 > > unless someone takes the initiative to fix it before v3 hits store > > shelves. This makes JDK9 the likely target for Apache Hadoop v4. > > > > Having major release cadences tied to JRE updates isn't > > necessarily a bad thing and definitely forces the community to a) > actually > > stop beating around the bush on majors and b) actually makes it > relatively > > easy to determine what the schedule looks like to some degree. > > > > > > > > > > > > - > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > >
[jira] [Created] (YARN-5638) Introduce a collector Id to uniquely identify collectors and their creation order
Li Lu created YARN-5638: --- Summary: Introduce a collector Id to uniquely identify collectors and their creation order Key: YARN-5638 URL: https://issues.apache.org/jira/browse/YARN-5638 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Li Lu Assignee: Li Lu As discussed in YARN-3359, we need to further identify timeline collectors and their creation order for better service discovery and resource isolation. This JIRA proposes to use to accurately identify each timeline collector. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5613) Fair Scheduler can assign containers from blacklisted nodes
[
https://issues.apache.org/jira/browse/YARN-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton resolved YARN-5613.
Resolution: Invalid
Turns out the issue I was seeing is coincidentally resolved by YARN-3547.
> Fair Scheduler can assign containers from blacklisted nodes
> ---
>
> Key: YARN-5613
> URL: https://issues.apache.org/jira/browse/YARN-5613
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-5613.001.patch
>
>
> The {{FairScheduler.allocate()}} makes its resource request before it updates
> the blacklist. If the scheduler processes the resource request before the
> allocating thread updates the blacklist, the scheduler can assign containers
> that are on nodes in the blacklist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5637) Changes in NodeManager to support Container upgrade and rollback/commit
Arun Suresh created YARN-5637: - Summary: Changes in NodeManager to support Container upgrade and rollback/commit Key: YARN-5637 URL: https://issues.apache.org/jira/browse/YARN-5637 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun Suresh Assignee: Arun Suresh YARN-5620 added support for re-initialization of Containers using a new launch Context. This JIRA proposes to use the above feature to support upgrade and subsequent rollback or commit of the upgrade. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[RESULTS] Re: [VOTE] Merge HADOOP-13341
The vote passes with 3 +1 binding votes. I'll be merging this later today. Thanks everyone! > On Sep 7, 2016, at 6:44 AM, Allen Wittenauer > wrote: > > > I’d like to call for a vote to run for 5 days (ending Mon 12, 2016 at > 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch was > developed exclusively by me. As usual with large shell script changes, it's > been broken up into several smaller commits to make it easier to read. The > core of the functionality is almost entirely in hadoop-functions.sh with the > majority of the rest of the new additions either being documentation or test > code. In addition, large swaths of code is removed from the hadoop, hdfs, > mapred, and yarn executables. > > Here's a quick summary: > > * makes the rules around _OPTS consistent across all the projects > * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, > and yarn subcommand > * with the exception of deprecations, removes all of the custom daemon _OPTS > handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands > * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it > consistent for non-daemon subcommands > * makes the _USER blocker consistent with _OPTS as well as providing better > documentation around this feature's existence. Note that this is an > incompatible change against -alpha1. > * by consolidating all of this code, makes it possible to finally fix a good > chunk of the "directory name containing spaces blows up the bash code" > problems that's been around since the beginning of the project > > Thanks! > > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/ No changes -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.datanode.TestBlockRecovery hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/162/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
