Re: [DISCUSS] More Maintenance Releases
I'm not sure we qualify as a big user, but FWIW, we are upgrading to a Hadoop based on 2.6.0 ( with our own application of critical bugfix patches that went in on branch-2 later ) over at Salesforce. 2.7.0 and up are scary because 2.7.0 was tagged as not ready for production. There's a may eat your data sign hanging over 2.7.0 and all subsequent releases and points along the branch-2 history. Even using 2.6.0 comes with some trepidation because we know the HDFS encryption feature added in that release will destroy any HBase installation if it is turned on. On Tue, Jun 23, 2015 at 11:55 AM, Karthik Kambatla ka...@cloudera.com wrote: On Tue, Jun 23, 2015 at 10:30 AM, Andrew Wang andrew.w...@cloudera.com wrote: There's no reason we have to choose 2.6 xor 2.7. If we have willing RMs and enough PMCs who will vote on releases, there's no reason we can't maintain both. However, based on the discussion at Hadoop Summit with Yahoo and Twitter, their interest is primarily in 2.6, and Daryn mentioned the need to get 2.6 stable before they can move to 2.7. So, if we want to help out these big users, it seems like we should focus on maintaining 2.6. It would be nice to hear from the big users on the topic. They are already on 2.6 and some of them have already done a bunch of backports. Given it will take us a month or two (based on past experience) to get the first release out, it is not clear to me if we should start focusing on 2.6 or 2.7. But again, as you say, if there are people willing to RM these releases, I see value in improving both lines. Allen also brought up the issue of JDK6. I see a few options (ranked best to worst in my eyes): * Add multi-JDK support to test-patch * Keep using JDK7 for precommit, and keep an eye on a nightly JDK6 run * Drop support for JDK6 in 2.6.x, since no one is using it anymore #3 is the most easiest and probably fine for 95% of users, but doing a big compat break is not how I'd want to kick off a stable release line. #2 isn't too bad if we don't want to wait for #1. Finally, an issue that Karthik mentioned at Hadoop Summit is that CHANGES.txt maintenance becomes a huge burden when doing backports after the initial commit. e.g. if I was backporting something to branch-2.6, I'd potentially need to update CHANGES.txt in trunk, branch-2, branch-2.7 to move the JIRA # to the right version. If we either automated or dropped CHANGES.txt, the process would be much smoother. +1 for ditching CHANGES.txt. I reached out to Allen recently. He wanted to clean up his scripts more before dropping CHANGES.txt altogether. Should we target 2.8 for this? Best, Andrew On Tue, Jun 23, 2015 at 6:05 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Thanks Tsuyoshi for the comment. Targeting to 2.7 looks good to me, however, I'd like to maintenance branch-2.6 also. It's only about a half year since 2.6.0 was released. I don't want to abandon relatively new branches. On 6/23/15 16:05, Tsuyoshi Ozawa wrote: Thank you for clarification, Karthik. I'd also like to work on stable release management with community. I'm thinking of speedy release against next version - 1 branch for making community feedback faster (e.g. current next version is 2.8, so cherry-picking to 2.7 and releasing it are useful work for users). I know that code base of 2.7.1 is now freezing currently, I'd love to work from 2.7.2 release if possible. Cheers, - Tsuyoshi On Tue, Jun 23, 2015 at 5:02 AM, Colin P. McCabe cmcc...@apache.org wrote: +1 for creating a maintenance release with a more rapid release cadence and more effort put into stability backports. I think this would really be great for the project. Colin On Mon, Jun 22, 2015 at 2:43 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira -- Karthik Kambatla Software Engineer, Cloudera Inc. http://five.sentenc.es --
Re: [DISCUSS] More Maintenance Releases
On Jun 26, 2015, at 3:35 PM, Andrew Wang andrew.w...@cloudera.com wrote: Allen, do you still have these concerns? You mentioned having written scripts to parse CHANGES.txt. I've written scripts for similar tasks, but what I turned to were git log and JIRA, not CHANGES.txt. I was wondering if this is a relic from the SVN days, since svn log was dog slow. Git log is quite a bit better. git log is totally unreliable. I’ve shown that over and over and over. JIRA may not perfect, but it’s *correctable* and the restraints on what actually can be input in certain fields is a good thing. But frankly, at this point, it doesn’t matter. We haven’t had a *single* minor release in branch-2 that didn’t break something in user- or ops- facing ways. Why stop now?
Re: [DISCUSS] More Maintenance Releases
+1 for ditching CHANGES.txt. I reached out to Allen recently. He wanted to clean up his scripts more before dropping CHANGES.txt altogether. Should we target 2.8 for this? I read through our previous threads on this topic, and Allen had some compatibility concerns about dropping CHANGES.txt in a branch-2 release. Allen, do you still have these concerns? You mentioned having written scripts to parse CHANGES.txt. I've written scripts for similar tasks, but what I turned to were git log and JIRA, not CHANGES.txt. I was wondering if this is a relic from the SVN days, since svn log was dog slow. Git log is quite a bit better. My inclination is to just drop CHANGES.txt entirely, including in branch-2. We already have release notes, JIRA, as well as the per-release webpage which talks up the new features. CHANGES.txt also has never been a reliable source of information, and I doubt it has many consumers (if any). Best, Andrew
Re: [DISCUSS] More Maintenance Releases
Thanks Tsuyoshi for the comment. Targeting to 2.7 looks good to me, however, I'd like to maintenance branch-2.6 also. It's only about a half year since 2.6.0 was released. I don't want to abandon relatively new branches. On 6/23/15 16:05, Tsuyoshi Ozawa wrote: Thank you for clarification, Karthik. I'd also like to work on stable release management with community. I'm thinking of speedy release against next version - 1 branch for making community feedback faster (e.g. current next version is 2.8, so cherry-picking to 2.7 and releasing it are useful work for users). I know that code base of 2.7.1 is now freezing currently, I'd love to work from 2.7.2 release if possible. Cheers, - Tsuyoshi On Tue, Jun 23, 2015 at 5:02 AM, Colin P. McCabe cmcc...@apache.org wrote: +1 for creating a maintenance release with a more rapid release cadence and more effort put into stability backports. I think this would really be great for the project. Colin On Mon, Jun 22, 2015 at 2:43 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira
Re: [DISCUSS] More Maintenance Releases
There's no reason we have to choose 2.6 xor 2.7. If we have willing RMs and enough PMCs who will vote on releases, there's no reason we can't maintain both. However, based on the discussion at Hadoop Summit with Yahoo and Twitter, their interest is primarily in 2.6, and Daryn mentioned the need to get 2.6 stable before they can move to 2.7. So, if we want to help out these big users, it seems like we should focus on maintaining 2.6. Allen also brought up the issue of JDK6. I see a few options (ranked best to worst in my eyes): * Add multi-JDK support to test-patch * Keep using JDK7 for precommit, and keep an eye on a nightly JDK6 run * Drop support for JDK6 in 2.6.x, since no one is using it anymore #3 is the most easiest and probably fine for 95% of users, but doing a big compat break is not how I'd want to kick off a stable release line. #2 isn't too bad if we don't want to wait for #1. Finally, an issue that Karthik mentioned at Hadoop Summit is that CHANGES.txt maintenance becomes a huge burden when doing backports after the initial commit. e.g. if I was backporting something to branch-2.6, I'd potentially need to update CHANGES.txt in trunk, branch-2, branch-2.7 to move the JIRA # to the right version. If we either automated or dropped CHANGES.txt, the process would be much smoother. Best, Andrew On Tue, Jun 23, 2015 at 6:05 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Thanks Tsuyoshi for the comment. Targeting to 2.7 looks good to me, however, I'd like to maintenance branch-2.6 also. It's only about a half year since 2.6.0 was released. I don't want to abandon relatively new branches. On 6/23/15 16:05, Tsuyoshi Ozawa wrote: Thank you for clarification, Karthik. I'd also like to work on stable release management with community. I'm thinking of speedy release against next version - 1 branch for making community feedback faster (e.g. current next version is 2.8, so cherry-picking to 2.7 and releasing it are useful work for users). I know that code base of 2.7.1 is now freezing currently, I'd love to work from 2.7.2 release if possible. Cheers, - Tsuyoshi On Tue, Jun 23, 2015 at 5:02 AM, Colin P. McCabe cmcc...@apache.org wrote: +1 for creating a maintenance release with a more rapid release cadence and more effort put into stability backports. I think this would really be great for the project. Colin On Mon, Jun 22, 2015 at 2:43 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira
Re: [DISCUSS] More Maintenance Releases
On Tue, Jun 23, 2015 at 10:30 AM, Andrew Wang andrew.w...@cloudera.com wrote: There's no reason we have to choose 2.6 xor 2.7. If we have willing RMs and enough PMCs who will vote on releases, there's no reason we can't maintain both. However, based on the discussion at Hadoop Summit with Yahoo and Twitter, their interest is primarily in 2.6, and Daryn mentioned the need to get 2.6 stable before they can move to 2.7. So, if we want to help out these big users, it seems like we should focus on maintaining 2.6. It would be nice to hear from the big users on the topic. They are already on 2.6 and some of them have already done a bunch of backports. Given it will take us a month or two (based on past experience) to get the first release out, it is not clear to me if we should start focusing on 2.6 or 2.7. But again, as you say, if there are people willing to RM these releases, I see value in improving both lines. Allen also brought up the issue of JDK6. I see a few options (ranked best to worst in my eyes): * Add multi-JDK support to test-patch * Keep using JDK7 for precommit, and keep an eye on a nightly JDK6 run * Drop support for JDK6 in 2.6.x, since no one is using it anymore #3 is the most easiest and probably fine for 95% of users, but doing a big compat break is not how I'd want to kick off a stable release line. #2 isn't too bad if we don't want to wait for #1. Finally, an issue that Karthik mentioned at Hadoop Summit is that CHANGES.txt maintenance becomes a huge burden when doing backports after the initial commit. e.g. if I was backporting something to branch-2.6, I'd potentially need to update CHANGES.txt in trunk, branch-2, branch-2.7 to move the JIRA # to the right version. If we either automated or dropped CHANGES.txt, the process would be much smoother. +1 for ditching CHANGES.txt. I reached out to Allen recently. He wanted to clean up his scripts more before dropping CHANGES.txt altogether. Should we target 2.8 for this? Best, Andrew On Tue, Jun 23, 2015 at 6:05 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Thanks Tsuyoshi for the comment. Targeting to 2.7 looks good to me, however, I'd like to maintenance branch-2.6 also. It's only about a half year since 2.6.0 was released. I don't want to abandon relatively new branches. On 6/23/15 16:05, Tsuyoshi Ozawa wrote: Thank you for clarification, Karthik. I'd also like to work on stable release management with community. I'm thinking of speedy release against next version - 1 branch for making community feedback faster (e.g. current next version is 2.8, so cherry-picking to 2.7 and releasing it are useful work for users). I know that code base of 2.7.1 is now freezing currently, I'd love to work from 2.7.2 release if possible. Cheers, - Tsuyoshi On Tue, Jun 23, 2015 at 5:02 AM, Colin P. McCabe cmcc...@apache.org wrote: +1 for creating a maintenance release with a more rapid release cadence and more effort put into stability backports. I think this would really be great for the project. Colin On Mon, Jun 22, 2015 at 2:43 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira -- Karthik Kambatla Software Engineer, Cloudera Inc. http://five.sentenc.es
Re: [DISCUSS] More Maintenance Releases
On Tue, Jun 23, 2015 at 12:30 PM, Andrew Wang andrew.w...@cloudera.com wrote: There's no reason we have to choose 2.6 xor 2.7. If we have willing RMs and enough PMCs who will vote on releases, there's no reason we can't maintain both. However, based on the discussion at Hadoop Summit with Yahoo and Twitter, their interest is primarily in 2.6, and Daryn mentioned the need to get 2.6 stable before they can move to 2.7. So, if we want to help out these big users, it seems like we should focus on maintaining 2.6. Allen also brought up the issue of JDK6. I see a few options (ranked best to worst in my eyes): * Add multi-JDK support to test-patch * Keep using JDK7 for precommit, and keep an eye on a nightly JDK6 run * Drop support for JDK6 in 2.6.x, since no one is using it anymore #3 is the most easiest and probably fine for 95% of users, but doing a big compat break is not how I'd want to kick off a stable release line. #2 isn't too bad if we don't want to wait for #1. I believe work on adding multi jdk is on-going for patch test. Should be high on the list once we get our dev branch up. -- Sean
Re: [DISCUSS] More Maintenance Releases
Thank you for clarification, Karthik. I'd also like to work on stable release management with community. I'm thinking of speedy release against next version - 1 branch for making community feedback faster (e.g. current next version is 2.8, so cherry-picking to 2.7 and releasing it are useful work for users). I know that code base of 2.7.1 is now freezing currently, I'd love to work from 2.7.2 release if possible. Cheers, - Tsuyoshi On Tue, Jun 23, 2015 at 5:02 AM, Colin P. McCabe cmcc...@apache.org wrote: +1 for creating a maintenance release with a more rapid release cadence and more effort put into stability backports. I think this would really be great for the project. Colin On Mon, Jun 22, 2015 at 2:43 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira
Re: [DISCUSS] More Maintenance Releases
Thanks Allen for reminding me of supporting JDK6. If 2.6 is the target, any commmiter needs to check a bug fix patch when cherry-picking it. For trunk, I'm thinking we need to decide what we should do for release 3.x, in another thread. I'm okay to discuss it again. On 6/23/15 01:19, Allen Wittenauer wrote: If 2.6 is the target, someone will have to verify that any cherry-picked patches actually work with JDK6 since the PMC voted to officially kill backward compatibility in a minor release. It’s going to be easier and probably smarter to fix 2.7 if that’s really desired. [1] Frankly, I’d rather see effort spent on stabilizing trunk and ditching the now broken branch-2. We’re approaching the 4 year anniversary of 0.23.0’s release (which later begat 2.x, which is already past the 3 year mark). It’s hard to claim health when its been so long since a branch off of trunk was cut and turned into something official. [1] Kengo and I are hard at work getting multiJDK testing working in Yetus, but it’s not quite ready for prime time. :( It could certain help here, but… it’s not very stable yet. On Jun 22, 2015, at 7:50 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for starting this thread, Akira. +1 to more maintenance releases. More stable upstream releases avoids duplicating cherry-pick work across consumers/vendors, and shows the maturity of the project to users. I see value in backporting blocker/critical issues, but have mixed feelings about doing the same for major/minor/trivial issues. IMO, every commit has non-zero potential to introduce other bugs. Depending on the kind of fix (say, documentation), it might be okay to include these non-critical fixes. One approach could be to allow all bug fixes for 2.x.1, blocker/critical for 2.x.2, blocker for 2.x.3 (or something along those lines) to ensure increasing stability of maintenance releases? I am also +1 to any committer picking up RM duties for a maintenance release. It is healthy to have more people participate in the release process, so long as we have some method to maintenance release madness. A committer (who is not yet a PMC member) could be a Release Manager, but his vote is not binding for the release. I RM-ed the 2.5.x releases as a committer. RM-ing a release and voting non-binding could be a good way to remind the PMC to include the committer in PMC :) Cheers Karthik On Mon, Jun 22, 2015 at 4:36 AM, Tsuyoshi Ozawa oz...@apache.org wrote: Hi Akira, Thank you for starting interesting topic. +1 on the idea of More Maintenance Releases for old branches. It would be good if this activity is more coupled with Apache Yetus for users. BTW, I don't know one of committers, who is not PMC, can be a release manager. Does anyone know about this? It's described in detail as follows: http://hadoop.apache.org/bylaws#Decision+Making Release Manager A Release Manager (RM) is a committer who volunteers to produce a Release Candidate according to HowToRelease. Project Management Committee Deciding what is distributed as products of the Apache Hadoop project. In particular all releases must be approved by the PMC Thanks, - Tsuyoshi On Mon, Jun 22, 2015 at 6:43 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira -- Karthik Kambatla Software Engineer, Cloudera Inc. http://five.sentenc.es
Re: [DISCUSS] More Maintenance Releases
Thanks Karthik for the comment. I'm +1 for your approach to ensure stability. On 6/22/15 23:50, Karthik Kambatla wrote: Thanks for starting this thread, Akira. +1 to more maintenance releases. More stable upstream releases avoids duplicating cherry-pick work across consumers/vendors, and shows the maturity of the project to users. I see value in backporting blocker/critical issues, but have mixed feelings about doing the same for major/minor/trivial issues. IMO, every commit has non-zero potential to introduce other bugs. Depending on the kind of fix (say, documentation), it might be okay to include these non-critical fixes. One approach could be to allow all bug fixes for 2.x.1, blocker/critical for 2.x.2, blocker for 2.x.3 (or something along those lines) to ensure increasing stability of maintenance releases? I am also +1 to any committer picking up RM duties for a maintenance release. It is healthy to have more people participate in the release process, so long as we have some method to maintenance release madness. A committer (who is not yet a PMC member) could be a Release Manager, but his vote is not binding for the release. I RM-ed the 2.5.x releases as a committer. RM-ing a release and voting non-binding could be a good way to remind the PMC to include the committer in PMC :) Cheers Karthik On Mon, Jun 22, 2015 at 4:36 AM, Tsuyoshi Ozawa oz...@apache.org wrote: Hi Akira, Thank you for starting interesting topic. +1 on the idea of More Maintenance Releases for old branches. It would be good if this activity is more coupled with Apache Yetus for users. BTW, I don't know one of committers, who is not PMC, can be a release manager. Does anyone know about this? It's described in detail as follows: http://hadoop.apache.org/bylaws#Decision+Making Release Manager A Release Manager (RM) is a committer who volunteers to produce a Release Candidate according to HowToRelease. Project Management Committee Deciding what is distributed as products of the Apache Hadoop project. In particular all releases must be approved by the PMC Thanks, - Tsuyoshi On Mon, Jun 22, 2015 at 6:43 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira
Re: [DISCUSS] More Maintenance Releases
If 2.6 is the target, someone will have to verify that any cherry-picked patches actually work with JDK6 since the PMC voted to officially kill backward compatibility in a minor release. It’s going to be easier and probably smarter to fix 2.7 if that’s really desired. [1] Frankly, I’d rather see effort spent on stabilizing trunk and ditching the now broken branch-2. We’re approaching the 4 year anniversary of 0.23.0’s release (which later begat 2.x, which is already past the 3 year mark). It’s hard to claim health when its been so long since a branch off of trunk was cut and turned into something official. [1] Kengo and I are hard at work getting multiJDK testing working in Yetus, but it’s not quite ready for prime time. :( It could certain help here, but… it’s not very stable yet. On Jun 22, 2015, at 7:50 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for starting this thread, Akira. +1 to more maintenance releases. More stable upstream releases avoids duplicating cherry-pick work across consumers/vendors, and shows the maturity of the project to users. I see value in backporting blocker/critical issues, but have mixed feelings about doing the same for major/minor/trivial issues. IMO, every commit has non-zero potential to introduce other bugs. Depending on the kind of fix (say, documentation), it might be okay to include these non-critical fixes. One approach could be to allow all bug fixes for 2.x.1, blocker/critical for 2.x.2, blocker for 2.x.3 (or something along those lines) to ensure increasing stability of maintenance releases? I am also +1 to any committer picking up RM duties for a maintenance release. It is healthy to have more people participate in the release process, so long as we have some method to maintenance release madness. A committer (who is not yet a PMC member) could be a Release Manager, but his vote is not binding for the release. I RM-ed the 2.5.x releases as a committer. RM-ing a release and voting non-binding could be a good way to remind the PMC to include the committer in PMC :) Cheers Karthik On Mon, Jun 22, 2015 at 4:36 AM, Tsuyoshi Ozawa oz...@apache.org wrote: Hi Akira, Thank you for starting interesting topic. +1 on the idea of More Maintenance Releases for old branches. It would be good if this activity is more coupled with Apache Yetus for users. BTW, I don't know one of committers, who is not PMC, can be a release manager. Does anyone know about this? It's described in detail as follows: http://hadoop.apache.org/bylaws#Decision+Making Release Manager A Release Manager (RM) is a committer who volunteers to produce a Release Candidate according to HowToRelease. Project Management Committee Deciding what is distributed as products of the Apache Hadoop project. In particular all releases must be approved by the PMC Thanks, - Tsuyoshi On Mon, Jun 22, 2015 at 6:43 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira -- Karthik Kambatla Software Engineer, Cloudera Inc. http://five.sentenc.es
Re: [DISCUSS] More Maintenance Releases
+1 for the idea of maintenance releases. Considering the amount code changes done in trunk and branch-2, cherry-picking may not be easy and straight forward in all issues. I would love to help in cherry-picking the fixes and reviewing them. I would also love to help in release process. Regards, Vinay On Mon, Jun 22, 2015 at 9:49 PM, Allen Wittenauer a...@altiscale.com wrote: If 2.6 is the target, someone will have to verify that any cherry-picked patches actually work with JDK6 since the PMC voted to officially kill backward compatibility in a minor release. It’s going to be easier and probably smarter to fix 2.7 if that’s really desired. [1] Frankly, I’d rather see effort spent on stabilizing trunk and ditching the now broken branch-2. We’re approaching the 4 year anniversary of 0.23.0’s release (which later begat 2.x, which is already past the 3 year mark). It’s hard to claim health when its been so long since a branch off of trunk was cut and turned into something official. [1] Kengo and I are hard at work getting multiJDK testing working in Yetus, but it’s not quite ready for prime time. :( It could certain help here, but… it’s not very stable yet. On Jun 22, 2015, at 7:50 AM, Karthik Kambatla ka...@cloudera.com wrote: Thanks for starting this thread, Akira. +1 to more maintenance releases. More stable upstream releases avoids duplicating cherry-pick work across consumers/vendors, and shows the maturity of the project to users. I see value in backporting blocker/critical issues, but have mixed feelings about doing the same for major/minor/trivial issues. IMO, every commit has non-zero potential to introduce other bugs. Depending on the kind of fix (say, documentation), it might be okay to include these non-critical fixes. One approach could be to allow all bug fixes for 2.x.1, blocker/critical for 2.x.2, blocker for 2.x.3 (or something along those lines) to ensure increasing stability of maintenance releases? I am also +1 to any committer picking up RM duties for a maintenance release. It is healthy to have more people participate in the release process, so long as we have some method to maintenance release madness. A committer (who is not yet a PMC member) could be a Release Manager, but his vote is not binding for the release. I RM-ed the 2.5.x releases as a committer. RM-ing a release and voting non-binding could be a good way to remind the PMC to include the committer in PMC :) Cheers Karthik On Mon, Jun 22, 2015 at 4:36 AM, Tsuyoshi Ozawa oz...@apache.org wrote: Hi Akira, Thank you for starting interesting topic. +1 on the idea of More Maintenance Releases for old branches. It would be good if this activity is more coupled with Apache Yetus for users. BTW, I don't know one of committers, who is not PMC, can be a release manager. Does anyone know about this? It's described in detail as follows: http://hadoop.apache.org/bylaws#Decision+Making Release Manager A Release Manager (RM) is a committer who volunteers to produce a Release Candidate according to HowToRelease. Project Management Committee Deciding what is distributed as products of the Apache Hadoop project. In particular all releases must be approved by the PMC Thanks, - Tsuyoshi On Mon, Jun 22, 2015 at 6:43 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira -- Karthik Kambatla Software Engineer, Cloudera Inc. http://five.sentenc.es
Re: [DISCUSS] More Maintenance Releases
Thanks for starting this thread, Akira. +1 to more maintenance releases. More stable upstream releases avoids duplicating cherry-pick work across consumers/vendors, and shows the maturity of the project to users. I see value in backporting blocker/critical issues, but have mixed feelings about doing the same for major/minor/trivial issues. IMO, every commit has non-zero potential to introduce other bugs. Depending on the kind of fix (say, documentation), it might be okay to include these non-critical fixes. One approach could be to allow all bug fixes for 2.x.1, blocker/critical for 2.x.2, blocker for 2.x.3 (or something along those lines) to ensure increasing stability of maintenance releases? I am also +1 to any committer picking up RM duties for a maintenance release. It is healthy to have more people participate in the release process, so long as we have some method to maintenance release madness. A committer (who is not yet a PMC member) could be a Release Manager, but his vote is not binding for the release. I RM-ed the 2.5.x releases as a committer. RM-ing a release and voting non-binding could be a good way to remind the PMC to include the committer in PMC :) Cheers Karthik On Mon, Jun 22, 2015 at 4:36 AM, Tsuyoshi Ozawa oz...@apache.org wrote: Hi Akira, Thank you for starting interesting topic. +1 on the idea of More Maintenance Releases for old branches. It would be good if this activity is more coupled with Apache Yetus for users. BTW, I don't know one of committers, who is not PMC, can be a release manager. Does anyone know about this? It's described in detail as follows: http://hadoop.apache.org/bylaws#Decision+Making Release Manager A Release Manager (RM) is a committer who volunteers to produce a Release Candidate according to HowToRelease. Project Management Committee Deciding what is distributed as products of the Apache Hadoop project. In particular all releases must be approved by the PMC Thanks, - Tsuyoshi On Mon, Jun 22, 2015 at 6:43 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira -- Karthik Kambatla Software Engineer, Cloudera Inc. http://five.sentenc.es
Re: [DISCUSS] More Maintenance Releases
Hi Akira, Thank you for starting interesting topic. +1 on the idea of More Maintenance Releases for old branches. It would be good if this activity is more coupled with Apache Yetus for users. BTW, I don't know one of committers, who is not PMC, can be a release manager. Does anyone know about this? It's described in detail as follows: http://hadoop.apache.org/bylaws#Decision+Making Release Manager A Release Manager (RM) is a committer who volunteers to produce a Release Candidate according to HowToRelease. Project Management Committee Deciding what is distributed as products of the Apache Hadoop project. In particular all releases must be approved by the PMC Thanks, - Tsuyoshi On Mon, Jun 22, 2015 at 6:43 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira
Re: [DISCUSS] More Maintenance Releases
+1 for creating a maintenance release with a more rapid release cadence and more effort put into stability backports. I think this would really be great for the project. Colin On Mon, Jun 22, 2015 at 2:43 AM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira
Re: [DISCUSS] More Maintenance Releases
More maintenance releases would be excellent. If y'all are going to make more releases on the 2.6 line, please consider backporting HADOOP-11710 as without it HBase is unusable on top of HDFS encryption. It's been inconvenient that the fix is only available in a non-production release line. -Sean On Mon, Jun 22, 2015 at 6:36 AM, Tsuyoshi Ozawa oz...@apache.org wrote: Hi Akira, Thank you for starting interesting topic. +1 on the idea of More Maintenance Releases for old branches. It would be good if this activity is more coupled with Apache Yetus for users. BTW, I don't know one of committers, who is not PMC, can be a release manager. Does anyone know about this? It's described in detail as follows: http://hadoop.apache.org/bylaws#Decision+Making Release Manager A Release Manager (RM) is a committer who volunteers to produce a Release Candidate according to HowToRelease. Project Management Committee Deciding what is distributed as products of the Apache Hadoop project. In particular all releases must be approved by the PMC Thanks, - Tsuyoshi On Mon, Jun 22, 2015 at 6:43 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Hi everyone, In Hadoop Summit, I joined HDFS BoF and heard from Jason Lowe that Apache Hadoop developers at Yahoo!, Twitter, and other non-distributors work very hard to maintenance Hadoop by cherry-picking patches to their own branches. I want to share the work with the community. If we can cherry-pick bug fix patches and have more maintenance releases, it'd be very happy not only for users but also for developers who work very hard for stabilizing their own branches. To have more maintenance releases, I propose two changes: * Major/Minor/Trivial bug fixes can be cherry-picked * (Roughly) Monthly maintenance release I would like to start the work from branch-2.6. If the change will be accepted by the community, I'm willing to work for the maintenance, as a release manager. Best regards, Akira -- Sean
