Re: [VOTE] Release Apache Hadoop 2.10.0 (RC1)

[Zhe Zhang]

+1 on RC1

- Verified signatures
- Built from source (RHEL7)
- Ran single node cluster and tested basic HDFS commands

Thanks for putting this together Jonathan!

On Mon, Oct 28, 2019 at 7:54 PM zhankun tang  wrote:

> Thanks for the efforts! Jonathan!
>
> +1 (non-binding) on RC1.
> - Set up a single node cluster with the binary tarball
> - Run Spark Pi and PySpark job
>
> BR,
> Zhankun
>
> On Mon, 28 Oct 2019 at 23:16, Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp>
> wrote:
>
> > Thanks for putting this up, Jonathan Hung.
> >
> > +1(non-binding)
> >
> > * verified signature and checksum.
> > * built source tarball by OpenJDK 7 on CentOS 7 with native profile
> > enabled.
> > * launched pseudo distributed cluster with security configurations.
> > * create encryption zone and put files in it.
> > * accessed files in encryption zone via httpfs.
> > * checked that file names containing '%' or '+' or ';' can be handled by
> > webhdfs.
> >
> > Masatake Iwasaki
> >
> > On 10/23/19 06:55, Jonathan Hung wrote:
> > > Hi folks,
> > >
> > > This is the second release candidate for the first release of Apache
> > Hadoop
> > > 2.10 line. It contains 362 fixes/improvements since 2.9 [1]. It
> includes
> > > features such as:
> > >
> > > - User-defined resource types
> > > - Native GPU support as a schedulable resource type
> > > - Consistent reads from standby node
> > > - Namenode port based selective encryption
> > > - Improvements related to rolling upgrade support from 2.x to 3.x
> > > - Cost based fair call queue
> > >
> > > The RC1 artifacts are at:
> > http://home.apache.org/~jhung/hadoop-2.10.0-RC1/
> > >
> > > RC tag is release-2.10.0-RC1.
> > >
> > > The maven artifacts are hosted here:
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1243/
> > >
> > > My public key is available here:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > The vote will run for 5 weekdays, until Tuesday, October 29 at 3:00 pm
> > PDT.
> > >
> > > Thanks,
> > > Jonathan Hung
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.0%20AND%20fixVersion%20not%20in%20(2.9.2%2C%202.9.1%2C%202.9.0)
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Zhe Zhang]

My late +1. Me and my colleagues have been involved in Submarine
development (in collaboration with our TonY project). I think Submarine
will be valuable as a new Apache project.

On Fri, Sep 6, 2019 at 10:57 PM Dinesh Chitlangia
 wrote:

> +1
>
> -Dinesh
>
>
>
>
> On Fri, Sep 6, 2019 at 11:23 PM 俊平堵  wrote:
>
> > +1. Please include me also.
> >
> > Thanks,
> >
> > Junping
> >
> > Wangda Tan  于2019年9月1日周日 下午1:19写道：
> >
> > > Hi all,
> > >
> > > As we discussed in the previous thread [1],
> > >
> > > I just moved the spin-off proposal to CWIKI and completed all TODO
> parts.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
> > >
> > > If you have interests to learn more about this. Please review the
> > proposal
> > > let me know if you have any questions/suggestions for the proposal.
> This
> > > will be sent to board post voting passed. (And please note that the
> > > previous voting thread [2] to move Submarine to a separate Github repo
> > is a
> > > necessary effort to move Submarine to a separate Apache project but not
> > > sufficient so I sent two separate voting thread.)
> > >
> > > Please let me know if I missed anyone in the proposal, and reply if
> you'd
> > > like to be included in the project.
> > >
> > > This voting runs for 7 days and will be concluded at Sep 7th, 11 PM
> PDT.
> > >
> > > Thanks,
> > > Wangda Tan
> > >
> > > [1]
> > >
> > >
> >
> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
> > > [2]
> > >
> > >
> >
> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
> > >
> >
>

Re: [VOTE] Merge YARN-8200 to branch-2 and branch-3.0

[Zhe Zhang]

+1 (binding)

As Jonathan said this feature in branch-2 has been running stably for over
a year. Therefore I’m supportive to the merge

On Mon, Aug 26, 2019 at 2:00 PM Jim Brennan 
wrote:

> +1 (non-binding).
> I have built branch-2 with the latest YARN-8200 patch
> (YARN-8200-branch-2.003.patch).  I ran all of the NM/RM tests and ran a few
> test jobs on a one-node cluster with default settings.
>
>
> On Mon, Aug 26, 2019 at 3:51 PM Oliver Hu  wrote:
>
> > +1 (non-binding)
> >
> > We have used this patch internally for more than a year to acquire GPU
> > reliably at LinkedIn. I don't think it is necessary to merge this to
> > branch-3.0 tho, as we are EOLing that branch.
> >
> > On Thu, Aug 22, 2019 at 4:43 PM Jonathan Hung 
> > wrote:
> >
> > > Hi folks,
> > >
> > > As per [1], starting a vote to merge YARN-8200 (and YARN-8200.branch3)
> > > feature branch to branch-2 (and branch-3.0).
> > >
> > > Vote runs for 7 days, to Thursday, Aug 29 5:00PM PDT.
> > >
> > > Thanks.
> > >
> > > [1]
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAHzWLgcX7f5Tr3q=csrqgysvpdf7mh-iu17femgx89dhr+1...@mail.gmail.com%3e
> > >
> > > Jonathan Hung
> > >
> >
>

Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

[Zhe Zhang]

+1

On Mon, Feb 4, 2019 at 6:14 PM Jonathan Hung  wrote:

> Hello,
>
> Starting a vote based on the discuss thread [1] for moving branch-2
> precommit/nightly test builds to openjdk8. After this change, the test
> phase for precommit builds [2] and branch-2 nightly build [3] will run on
> openjdk8. To maintain source compatibility, these builds will still run
> their compile phase for branch-2 on openjdk7 as they do now (in addition to
> compiling on openjdk8).
>
> Vote will run for three business days until Thursday Feb 7 6:00PM PDT.
>
> [1]
>
> https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E
>
> [2]
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/
>
> [3]
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
>
> Jonathan Hung
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Propose to start new Hadoop sub project "submarine"

[Zhe Zhang]

+1, binding

On Mon, Feb 4, 2019 at 1:20 PM Takanobu Asanuma 
wrote:

> +1 (non-binding). Thanks, Wangda!
>
> - Takanobu
>
> on 2019/02/02 7:24, "Wangda Tan" wrote:
>
> Hi all,
>
> According to positive feedbacks from the thread [1]
>
> This is vote thread to start a new subproject named "hadoop-submarine"
> which follows the release process already established for ozone.
>
> The vote runs for usual 7 days, which ends at Feb 8th 5 PM PDT.
>
> Thanks,
> Wangda Tan
>
> [1]
>
> https://lists.apache.org/thread.html/f864461eb188bd12859d51b0098ec38942c4429aae7e4d001a633d96@%3Cyarn-dev.hadoop.apache.org%3E
>
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [DISCUSS] Making submarine to different release model like Ozone

[Zhe Zhang]

+1 on the proposal and looking forward to the progress of the project!

On Thu, Jan 31, 2019 at 10:51 PM Weiwei Yang  wrote:

> Thanks for proposing this Wangda, my +1 as well.
> It is amazing to see the progress made in Submarine last year, the
> community grows fast and quiet collaborative. I can see the reasons to get
> it release faster in its own cycle. And at the same time, the Ozone way
> works very well.
>
> —
> Weiwei
> On Feb 1, 2019, 10:49 AM +0800, Xun Liu , wrote:
> > +1
> >
> > Hello everyone,
> >
> > I am Xun Liu, the head of the machine learning team at Netease Research
> Institute. I quite agree with Wangda.
> >
> > Our team is very grateful for getting Submarine machine learning engine
> from the community.
> > We are heavy users of Submarine.
> > Because Submarine fits into the direction of our big data team's hadoop
> technology stack,
> > It avoids the needs to increase the manpower investment in learning
> other container scheduling systems.
> > The important thing is that we can use a common YARN cluster to run
> machine learning,
> > which makes the utilization of server resources more efficient, and
> reserves a lot of human and material resources in our previous years.
> >
> > Our team have finished the test and deployment of the Submarine and will
> provide the service to our e-commerce department (http://www.kaola.com/)
> shortly.
> >
> > We also plan to provides the Submarine engine in our existing YARN
> cluster in the next six months.
> > Because we have a lot of product departments need to use machine
> learning services,
> > for example:
> > 1) Game department (http://game.163.com/) needs AI battle training,
> > 2) News department (http://www.163.com) needs news recommendation,
> > 3) Mailbox department (http://www.163.com) requires anti-spam and
> illegal detection,
> > 4) Music department (https://music.163.com/) requires music
> recommendation,
> > 5) Education department (http://www.youdao.com) requires voice
> recognition,
> > 6) Massive Open Online Courses (https://open.163.com/) requires
> multilingual translation and so on.
> >
> > If Submarine can be released independently like Ozone, it will help us
> quickly get the latest features and improvements, and it will be great
> helpful to our team and users.
> >
> > Thanks hadoop Community!
> >
> >
> > > 在 2019年2月1日，上午2:53，Wangda Tan  写道：
> > >
> > > Hi devs,
> > >
> > > Since we started submarine-related effort last year, we received a lot
> of
> > > feedbacks, several companies (such as Netease, China Mobile, etc.) are
> > > trying to deploy Submarine to their Hadoop cluster along with big data
> > > workloads. Linkedin also has big interests to contribute a Submarine
> TonY (
> > > https://github.com/linkedin/TonY) runtime to allow users to use the
> same
> > > interface.
> > >
> > > From what I can see, there're several issues of putting Submarine under
> > > yarn-applications directory and have same release cycle with Hadoop:
> > >
> > > 1) We started 3.2.0 release at Sep 2018, but the release is done at Jan
> > > 2019. Because of non-predictable blockers and security issues, it got
> > > delayed a lot. We need to iterate submarine fast at this point.
> > >
> > > 2) We also see a lot of requirements to use Submarine on older Hadoop
> > > releases such as 2.x. Many companies may not upgrade Hadoop to 3.x in a
> > > short time, but the requirement to run deep learning is urgent to
> them. We
> > > should decouple Submarine from Hadoop version.
> > >
> > > And why we wanna to keep it within Hadoop? First, Submarine included
> some
> > > innovation parts such as enhancements of user experiences for YARN
> > > services/containerization support which we can add it back to Hadoop
> later
> > > to address common requirements. In addition to that, we have a big
> overlap
> > > in the community developing and using it.
> > >
> > > There're several proposals we have went through during Ozone merge to
> trunk
> > > discussion:
> > >
> https://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201803.mbox/%3ccahfhakh6_m3yldf5a2kq8+w-5fbvx5ahfgs-x1vajw8gmnz...@mail.gmail.com%3E
> > >
> > > I propose to adopt Ozone model: which is the same master branch,
> different
> > > release cycle, and different release branch. It is a great example to
> show
> > > agile release we can do (2 Ozone releases after Oct 2018) with less
> > > overhead to setup CI, projects, etc.
> > >
> > > *Links:*
> > > - JIRA: https://issues.apache.org/jira/browse/YARN-8135
> > > - Design doc
> > > <
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit
> >
> > > - User doc
> > > <
> https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/Index.html
> >
> > > (3.2.0
> > > release)
> > > - Blogposts, {Submarine} : Running deep learning workloads on Apache
> Hadoop
> > > <
> https://hortonworks.com/blog/submarine-running-deep-learning-workloads-apache-hadoop/
> >,
> > > (Chinese 

Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Zhe Zhang]

+1

Thanks for addressing concerns from the previous vote.

On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko 
wrote:

> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Zhe Zhang]

+1 (binding)

Thanks Konstantin for leading the merge effort!

I worked very closely with Chen, Konstantin, and Erik in the testing stage
and I feel confident that the feature has now completed designed
functionalities and has proven to be stable.

Great team work with contributors from multiple companies!

On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko 
wrote:

> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> There are a few outstanding issues:
> 1. Need to provide proper documentation - a user guide for the new feature
> 2. Need to fix automatic failover with ZKFC. Currently it does not doesn't
> know about ObserverNodes trying to convert them to SBNs.
> 3. Scale testing and performance fine-tuning
> 4. As testing progresses, we continue fixing non-critical bugs like
> HDFS-14116.
>
> I attached a unified patch to the umbrella jira for the review and Jenkins
> build.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 12.
>
> Thanks,
> --Konstantin
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 2.7.6 (RC0)

[Zhe Zhang]

+1 (binding)

- Downloaded source
- Verified checksum
- Built and started local cluster
- Checked YARN UI
- Performed basic HDFS operations

On Fri, Apr 13, 2018 at 1:40 PM Chen Liang <vagaryc...@gmail.com> wrote:

> Thanks for working on this Konstantin!
>
> +1 (non-binding)
>
> - verified checksum
> - built from source
> - started a single node HDFS cluster
> - performed basic operations of ls/mkdir/put/get
> - checked web UI
> - ran the MR job pi
>
> Regards,
> Chen
>
>
> > On Apr 9, 2018, at 4:14 PM, Konstantin Shvachko <shv.had...@gmail.com>
> wrote:
> >
> > Hi everybody,
> >
> > This is the next dot release of Apache Hadoop 2.7 line. The previous one
> 2.7.5
> > was released on December 14, 2017.
> > Release 2.7.6 includes critical bug fixes and optimizations. See more
> > details in Release Note:
> > http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html
> >
> > The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/
> >
> > Please give it a try and vote on this thread. The vote will run for 5
> days
> > ending 04/16/2018.
> >
> > My up to date public key is available from:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Thanks,
> > --Konstantin
>
>
> ---------
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

[jira] [Resolved] (YARN-6067) Applications API Service HA

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang resolved YARN-6067.
-
Resolution: Duplicate

> Applications API Service HA
> ---
>
> Key: YARN-6067
> URL: https://issues.apache.org/jira/browse/YARN-6067
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>
> We need to start thinking about HA for the Applications API Service. How do 
> we achieve it? Should API Service become part of the RM process to get a lot 
> of things for free? Should there be some other strategy. We need to start the 
> discussion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler configuration) to trunk, branch-3.0, branch-2

[Zhe Zhang]

+1 (binding)

LinkedIn has been running an internal version of OrgQueue for almost 2
years. It is an essential feature for running YARN scheduler on clusters
shared by multiple internal organizations.

I have participated in the YARN-5734 design and have regularly synced with
Jonathan on development and implementation details. I believe the feature
is of decent quality now and is ready to merge into trunk.

On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jyhung2...@gmail.com> wrote:

> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here
> (
> YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> <https://issues.apache.org/jira/browse/YARN-5949>) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=pkec9dpggyx9rzf3n...@mail.gmail.com%3E
>
> Jonathan Hung
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [RESULT] [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Zhe Zhang]

Thanks Konstantin, great work!

On Sat, Aug 5, 2017 at 1:36 PM Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> My formal vote
> +1 (binding)
>
> I am glad to summarize that
> with 7 binding and 13 non-binding +1s and no -1s the vote for Apache
> Release 2.7.4 passes.
> Thank you everybody for contributing to the release, testing it, and
> voting.
>
> Binding +1s (7)
> Zhe Zhang
> Jason Lowe
> Eric Payne
> Sunil G
> Akira Ajisaka
> Chris Douglas
> Konstantin Shvachko
>
> Non-binding +1s (13)
> John Zhuge
> Surendra Lilhore
> Masatake Iwasaki
> Hanisha Koneru
> Chen Liang
> Fnu Ajay Kumar
> Brahma Reddy Battula
> Edwina Lu
> Ye Zhou
> Eric Badger
> Mingliang Liu
> Kuhu Shukla
> Erik Krogen
>
> Thanks,
> --Konstantin
>
> On Sat, Jul 29, 2017 at 4:29 PM, Konstantin Shvachko <shv.had...@gmail.com
> >
> wrote:
>
> > Hi everybody,
> >
> > Here is the next release of Apache Hadoop 2.7 line. The previous stable
> > release 2.7.3 was available since 25 August, 2016.
> > Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
> > critical bug fixes and major optimizations. See more details in Release
> > Note:
> > http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
> >
> > The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
> >
> > Please give it a try and vote on this thread. The vote will run for 5
> days
> > ending 08/04/2017.
> >
> > Please note that my up to date public key are available from:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > Please don't forget to refresh the page if you've been there recently.
> > There are other place on Apache sites, which may contain my outdated key.
> >
> > Thanks,
> > --Konstantin
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Zhe Zhang]

Quick question for Chris: was your +1 for the RC, or to support
Konstantin's statement regarding packaging?

Thanks,

On Mon, Jul 31, 2017 at 3:56 PM Chris Douglas <cdoug...@apache.org> wrote:

> On Mon, Jul 31, 2017 at 3:02 PM, Konstantin Shvachko
> <shv.had...@gmail.com> wrote:
> > For the packaging, here is the exact phrasing from the sited
> release-policy
> > document relevant to binaries:
> > "As a convenience to users that might not have the appropriate tools to
> > build a compiled version of the source, binary/bytecode packages MAY be
> > distributed alongside official Apache releases. In all such cases, the
> > binary/bytecode package MUST have the same version number as the source
> > release and MUST only add binary/bytecode files that are the result of
> > compiling that version of the source code release and its dependencies."
> > I don't think my binary package violates any of these.
>
> +1 The PMC VOTE applies to source code, only. If someone wants to
> rebuild the binary tarball with native libs and replace this one,
> that's fine.
>
> My reading of the above is that source code must be distributed with
> binaries, not that we omit the source code from binary releases... -C
>
> > But I'll upload an additional tar.gz with native bits and no src, as you
> > guys requested.
> > Will keep it as RC0 as there is no source code change and it comes from
> the
> > same build.
> > Hope this is satisfactory.
> >
> > Thanks,
> > --Konstantin
> >
> > On Mon, Jul 31, 2017 at 1:53 PM, Andrew Wang <andrew.w...@cloudera.com>
> > wrote:
> >
> >> I agree with Brahma on the two issues flagged (having src in the binary
> >> tarball, missing native libs). These are regressions from prior
> releases.
> >>
> >> As an aside, "we release binaries as a convenience" doesn't relax the
> >> quality bar. The binaries are linked on our website and distributed
> through
> >> official Apache channels. They have to adhere to Apache release
> >> requirements. And, most users consume our work via Maven dependencies,
> >> which are binary artifacts.
> >>
> >> http://www.apache.org/legal/release-policy.html goes into this in more
> >> detail. A release must minimally include source packages, and can also
> >> include binary artifacts.
> >>
> >> Best,
> >> Andrew
> >>
> >> On Mon, Jul 31, 2017 at 12:30 PM, Konstantin Shvachko <
> >> shv.had...@gmail.com> wrote:
> >>
> >>> To avoid any confusion in this regard. I built RC0 manually in
> compliance
> >>> with Apache release policy
> >>> http://www.apache.org/legal/release-policy.html
> >>> I edited the HowToReleasePreDSBCR page to make sure people don't use
> >>> Jenkins option for building.
> >>>
> >>> A side note. This particular build is broken anyways, so no worries
> there.
> >>> I think though it would be useful to have it working for testing and
> as a
> >>> packaging standard.
> >>>
> >>> Thanks,
> >>> --Konstantin
> >>>
> >>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
> >>> a...@effectivemachines.com
> >>> > wrote:
> >>>
> >>> >
> >>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
> >>> shv.had...@gmail.com>
> >>> > wrote:
> >>> > >
> >>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
> >>> >
> >>> > FYI:
> >>> >
> >>> > If you are using ASF Jenkins to create an ASF release
> >>> > artifact, it's pretty much an automatic vote failure as any such
> >>> release is
> >>> > in violation of ASF policy.
> >>> >
> >>> >
> >>>
> >>
> >>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Zhe Zhang]

gt; > > >>> I edited the HowToReleasePreDSBCR page to make sure people don't
> use
> > > >>> Jenkins option for building.
> > > >>>
> > > >>> A side note. This particular build is broken anyways, so no worries
> > > there.
> > > >>> I think though it would be useful to have it working for testing
> and
> > > as a
> > > >>> packaging standard.
> > > >>>
> > > >>> Thanks,
> > > >>> --Konstantin
> > > >>>
> > > >>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
> > > >>> a...@effectivemachines.com
> > > >>> > wrote:
> > > >>>
> > > >>> >
> > > >>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
> > > >>> shv.had...@gmail.com>
> > > >>> > wrote:
> > > >>> > >
> > > >>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
> > > >>> >
> > > >>> > FYI:
> > > >>> >
> > > >>> > If you are using ASF Jenkins to create an ASF
> > release
> > > >>> > artifact, it's pretty much an automatic vote failure as any such
> > > >>> release is
> > > >>> > in violation of ASF policy.
> > > >>> >
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > >
> >
>
>
>
> --
>
> *Zhou, Ye  **周晔*
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: About 2.7.4 Release

[Zhe Zhang]

Thanks for volunteering as RM Konstantin! The plan LGTM.

I've created a nightly Jenkins job for branch-2.7 (unit tests):
https://builds.apache.org/job/Hadoop-branch2.7-nightly/

On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko <shv.had...@gmail.com>
wrote:

> Hey guys,
>
> I and a few of my colleagues would like to help here and move 2.7.4 release
> forward. A few points in this regard.
>
> 1. Reading through this thread since March 1 I see that Vinod hinted on
> managing the release. Vinod, if you still want the job / have bandwidth
> will be happy to work with you.
> Otherwise I am glad to volunteer as the release manager.
>
> 2. In addition to current blockers and criticals, I would like to propose a
> few issues to be included in the release, see the list below. Those are
> mostly bug fixes and optimizations, which we already have in our internal
> branch and run in production. Plus one minor feature "node labeling", which
> we found very handy, when you have heterogeneous environments and mixed
> workloads, like MR and Spark.
>
> 3. For marking issues for the release I propose to
>  - set the target version to 2.7.4, and
>  - add a new label "release-blocker"
> That way we will know issues targeted for the release without reopening
> them for backports.
>
> 4. I see quite a few people are interested in the release. With all the
> help I think we can target to release by the end of May.
>
> Other things include fixing CHANGES.txt and fixing Jenkins build for 2.7.4
> branch.
>
> Thanks,
> --Konstantin
>
> ==  List of issue for 2.7.4  ===
> -- Backports
> HADOOP-12975 <https://issues.apache.org/jira/browse/HADOOP-12975>. Add du
> jitters
> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR batching
> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>. NPE when
> applying AvailableSpaceBlockPlacementPolicy
> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>. fsck removal
> of dot printing
> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
> space-balanced
> policy for balancer
> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>. abort
> balancer
> if upgrade in progress
> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>. skip small
> blocks in getBlocks
>
> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>. SLS simulator
> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>. SLS
> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>. SLS
> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>. SLS
>
> - Node labeling
> MAPREDUCE-6304 <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Jenkins patch-compile fails on trunk

[Zhe Zhang]

https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/patch-compile-root.txt

Looks related to yarn-ui? The HDFS-10872 patch doesn't touch YARN, but the
patch-compile failed at YARN.

Thanks,
Zhe

Re: Multiple node Hadoop cluster on Docker for test and debugging

[Zhe Zhang]

Nice work Kai! Quick comment: have you considered Bigtop? It allows
creating a multi-node docker cluster too. But I'm not sure if Bigtop takes
a custom built Hadoop (from the instructions I read, it builds from a
certain git branch; I'm not sure if that customizable).

On Wed, Nov 9, 2016 at 5:39 AM Sasaki Kai <lewua...@me.com> wrote:

> Hi Hadoop developers
>
> The other day I created a tool for launching multiple node hadoop cluster
> on docker container.
> You can easily launch multiple node hadoop cluster from your Hadoop source
> code.
> It is useful for testing and debugging. Actually I often use it before
> submitting a patch to Hadoop project.
> https://github.com/Lewuathe/docker-hadoop-cluster <
> https://github.com/Lewuathe/docker-hadoop-cluster>
>
> And I also updated to build the latest trunk image automatically and
> upload onto Docker Hub.
> So you can easily check and test the latest trunk branch in the
> environment which is more close to actual usage.
>
> If you already installed docker and docker-compose, what needed is
> docker-compose.yml like this.
>
> version: '2'
>
> services:
>   master:
> image: lewuathe/hadoop-master
> ports:
>   - "9870:9870"
>   - "8088:8088"
>   - "19888:19888"
>   - "8188:8188"
> container_name: "master"
>   slave1:
> image: lewuathe/hadoop-slave
> container_name: "slave1"
> depends_on:
>   - master
> ports:
>   - "9901:9864"
>   - "8041:8042"
>   slave2:
> image: lewuathe/hadoop-slave
> container_name: "slave2"
> depends_on:
>   - master
> ports:
>   - "9902:9864"
>   - "8042:8042"
>
> The usage in detail is described in the repository.
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md <
> https://github.com/Lewuathe/docker-hadoop-cluster/blob/master/README.md>
>
> I would be glad if you use this tool for developing and debugging and make
> our development more efficient.
> Please give me any feedbacks to me. Thanks you!
>
> 
> Kai Sasaki
> mail: lewua...@me.com <mailto:lewua...@me.com>
> github: https://github.com/Lewuathe <https://github.com/Lewuathe>
>
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

[jira] [Created] (YARN-5854) Throw more accurate exceptions from LinuxContainerExecutor#init

[Zhe Zhang (JIRA)]

Zhe Zhang created YARN-5854:
---

 Summary: Throw more accurate exceptions from 
LinuxContainerExecutor#init
 Key: YARN-5854
 URL: https://issues.apache.org/jira/browse/YARN-5854
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager, yarn
Reporter: Zhe Zhang
Assignee: Jonathan Hung
Priority: Minor


YARN-5822 logs {{ContainerExecutionException}}, but doesn't include exception 
{{e}} in the IOException it throws.

Another improvement is to reduce the duplicate exception messages:
# "Failed to bootstrap configured resource subsystems!"
# "Failed to initialize linux container runtime(s)!"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-4998) Minor cleanup to UGI use in AdminService

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang resolved YARN-4998.
-
Resolution: Fixed

> Minor cleanup to UGI use in AdminService
> 
>
> Key: YARN-4998
> URL: https://issues.apache.org/jira/browse/YARN-4998
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Attachments: YARN-4998.001.patch, YARN-4998.002.patch
>
>
> Instead of calling {{UserGroupInformation.getCurrentUser()}} over and over, 
> we should just use the stored {{daemonUser}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Updated 2.8.0-SNAPSHOT artifact

[Zhe Zhang]

I think recutting branch-2.8 from branch-2 is a good idea. We are running
2.6.x in production, and as a result we spend quite some effort for each
patch we want to backport (branch-2 => branch-2.8 => branch-2.7).

On Mon, Oct 31, 2016 at 1:20 AM Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> Thanks Akira and Karthik for replies.
>
>
> If there is no significant benefit over current branch-2.8 ,re-cutting
> will be an good idea, As we can reduce the maintenance effort.
>
>
>
> --Brahma Reddy Battula
>
>
> -Original Message-
> From: Akira Ajisaka [mailto:ajisa...@oss.nttdata.co.jp]
> Sent: 31 October 2016 14:06
> To: Karthik Kambatla
> Cc: Brahma Reddy Battula; Vinod Kumar Vavilapalli;
> common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: Re: Updated 2.8.0-SNAPSHOT artifact
>
> Thanks Karthik for the comment.
>
> Pros:
>
> * (Probably) we can release 2.8.0 earlier.
>
>* New feature in 2.8.0 will be available sooner.
>* We can deprecate APIs sooner. After 2.8.0 is released, we can remove
> them from trunk.
>
> Cons:
>
> * Maintenance cost becomes higher because # of branches becomes more.
> * New feature in 2.9.0 will be available later
>
> I'm okay with either releasing current branch-2.8 or re-cutting it.
>
> Thoughts?
>
> Regards,
> Akira
>
> On 10/26/16 07:30, Karthik Kambatla wrote:
> > Is there value in releasing current branch-2.8? Aren't we better off
> > re-cutting the branch off of branch-2?
> >
> > On Tue, Oct 25, 2016 at 12:20 AM, Akira Ajisaka
> > <ajisa...@oss.nttdata.co.jp>
> > wrote:
> >
> >> It's almost a year since branch-2.8 has cut.
> >> I'm thinking we need to release 2.8.0 ASAP.
> >>
> >> According to the following list, there are 5 blocker and 6 critical
> issues.
> >> https://issues.apache.org/jira/issues/?filter=12334985
> >>
> >> Regards,
> >> Akira
> >>
> >>
> >> On 10/18/16 10:47, Brahma Reddy Battula wrote:
> >>
> >>> Hi Vinod,
> >>>
> >>> Any plan on first RC for branch-2.8 ? I think, it has been long time.
> >>>
> >>>
> >>>
> >>>
> >>> --Brahma Reddy Battula
> >>>
> >>> -Original Message-
> >>> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org]
> >>> Sent: 20 August 2016 00:56
> >>> To: Jonathan Eagles
> >>> Cc: common-...@hadoop.apache.org
> >>> Subject: Re: Updated 2.8.0-SNAPSHOT artifact
> >>>
> >>> Jon,
> >>>
> >>> That is around the time when I branched 2.8, so I guess you were
> >>> getting SNAPSHOT artifacts till then from the branch-2 nightly builds.
> >>>
> >>> If you need it, we can set up SNAPSHOT builds. Or just wait for the
> >>> first RC, which is around the corner.
> >>>
> >>> +Vinod
> >>>
> >>> On Jul 28, 2016, at 4:27 PM, Jonathan Eagles <jeag...@gmail.com>
> wrote:
> >>>>
> >>>> Latest snapshot is uploaded in Nov 2015, but checkins are still
> >>>> coming in quite frequently.
> >>>> https://repository.apache.org/content/repositories/snapshots/org/ap
> >>>> ach
> >>>> e/hadoop/hadoop-yarn-api/
> >>>>
> >>>> Are there any plans to start producing updated SNAPSHOT artifacts
> >>>> for current hadoop development lines?
> >>>>
> >>>
> >>>
> >>> 
> >>> - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>>
> >>>
> >>> 
> >>> - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>>
> >>>
> >>
> >> -
> >> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> >>
> >>
> >
>
> --
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Zhe Zhang]

+1 (non-binding)

Did the following on 7 RHEL 6.6 servers
- Downloaded and built from source
- Downloaded and verified checksum of the binary tar.gz file
- Setup a cluster with 1 NN and 6 DNs
- Tried regular HDFS commands
- Tried EC commands (listPolicies, getPolicy, setPolicy), they work fine
- Verified that with a 3-2 policy, 1.67x capacity is used. Below is the
output after copying the binary tar.gz file into an EC folder. The file is
318MB.

Configured Capacity: 3221225472 (3 GB)
Present Capacity: 3215348743 (2.99 GB)
DFS Remaining: 2655666176 (2.47 GB)
DFS Used: 559682567 (533.75 MB)

Thanks Allen for clarifying on the markdown files. I also verified the site
html files (content of the index.html, randomly selected some links).


On Tue, Aug 30, 2016 at 2:20 PM Eric Badger 
wrote:

> Well that's embarrassing. I had accidentally slightly renamed my
> log4j.properties file in my conf directory, so it was there, just not being
> read. Apologies for the unnecessary spam. With this and the public key from
> Andrew, I give my non-binding +1.
>
> Eric
>
>
>
> On Tuesday, August 30, 2016 4:11 PM, Allen Wittenauer <
> a...@effectivemachines.com> wrote:
>
>
> > On Aug 30, 2016, at 2:06 PM, Eric Badger 
> wrote:
> >
> >
> > WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be
> incomplete.
>
> ^^
>
>
> >
> > After running the above command, the RM UI showed a successful job, but
> as you can see, I did not have anything printed onto the command line.
> Hopefully this is just a misconfiguration on my part, but I figured that I
> would point it out just in case.
>
>
> It gave you a very important message in the output ...
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Zhe Zhang]

Thanks Andrew for the great work! It's really exciting to finally see a
Hadoop 3 RC.

I noticed CHANGES and RELEASENOTES markdown files which were not in
previous RCs like 2.7.3. What are good tools to verify them? I tried
reading them on IntelliJ but format looks odd.

I'm still testing the RC:
- Downloaded and verified checksum
- Built from source
- Will start small cluster and test simple programs, focusing on EC
functionalities

-- Zhe

On Tue, Aug 30, 2016 at 8:51 AM Andrew Wang 
wrote:

> Hi all,
>
> Thanks to the combined work of many, many contributors, here's an RC0 for
> 3.0.0-alpha1:
>
> http://home.apache.org/~wang/3.0.0-alpha1-RC0/
>
> alpha1 is the first in a series of planned alpha releases leading up to GA.
> The objective is to get an artifact out to downstreams for testing and to
> iterate quickly based on their feedback. So, please keep that in mind when
> voting; hopefully most issues can be addressed by future alphas rather than
> future RCs.
>
> Sorry for getting this out on a Tuesday, but I'd still like this vote to
> run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend
> if we lack the votes.
>
> Please try it out and let me know what you think.
>
> Best,
> Andrew
>

Re: [DISCUSS] Increased use of feature branches

[Zhe Zhang]

Thanks for the notes Andrew, Junping, Karthik.

Here are some of my understandings:

- Trunk is the "latest and greatest" of Hadoop. If a user starts using
Hadoop today, without legacy workloads, trunk is what he/she should use.
- Therefore, each commit to trunk should be transactional -- atomic,
consistent, isolated (from other uncommitted patches); I'm not so sure
about durability, Hadoop might be gone in 50 years :). As a committer, I
should be able to look at a patch and determine whether it's a
self-contained improvement of trunk, without looking at other uncommitted
patches.
- Some comments inline:

On Fri, Jun 10, 2016 at 6:56 AM Junping Du  wrote:

> Comparing with advantages, I believe the disadvantages of shipping any
> releases directly from trunk are more obvious and significant:
> - A lot of commits (incompatible, risky, uncompleted feature, etc.) have
> to wait to commit to trunk or put into a separated branch that could delay
> feature development progress as additional vote process get involved even
> the feature is simple and harmless.
>
Thanks Junping, those are valid concerns. I think we should clearly
separate incompatible with  uncompleted / half-done work in this
discussion. Whether people should commit incompatible changes to trunk is a
much more tricky question (related to trunk-incompat etc.). But per my
comment above, IMHO, *not committing uncompleted work to trunk* should be a
much easier principle to agree upon.


> - For small feature with only 1 or 2 commits, that need three +1 from PMCs
> will increase the bar largely for contributors who just start to contribute
> on Hadoop features but no such sufficient support.
>
Development overhead is another valid concern. I think our rule-of-thumb
should be that, small-medium new features should be proposed as a single
JIRA/patch (as we recently did for HADOOP-12666). If the complexity goes
beyond a single JIRA/patch, use a feature branch.


>
> Given these concerns, I am open to other options, like: proposed by Vinod
> or Chris, but rather than to release anything directly from trunk.
>
> - This point doesn't necessarily need to be resolved now though, since
> again we're still doing alphas.
> No. I think we have to settle down this first. Without a common agreed and
> transparent release process and branches in community, any release (alpha,
> beta) bits is only called a private release but not a official apache
> hadoop release (even alpha).
>
>
> Thanks,
>
> Junping
> 
> From: Karthik Kambatla 
> Sent: Friday, June 10, 2016 7:49 AM
> To: Andrew Wang
> Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: Re: [DISCUSS] Increased use of feature branches
>
> Thanks for restarting this thread Andrew. I really hope we can get this
> across to a VOTE so it is clear.
>
> I see a few advantages shipping from trunk:
>
>- The lack of need for one additional backport each time.
>- Feature rot in trunk
>
> Instead of creating branch-3, I recommend creating branch-3.x so we can
> continue doing 3.x releases off branch-3 even after we move trunk to 4.x (I
> said it :))
>
> On Thu, Jun 9, 2016 at 11:12 PM, Andrew Wang 
> wrote:
>
> > Hi all,
> >
> > On a separate thread, a question was raised about 3.x branching and use
> of
> > feature branches going forward.
> >
> > We discussed this previously on the "Looking to a Hadoop 3 release"
> thread
> > that has spanned the years, with Vinod making this proposal (building on
> > ideas from others who also commented in the email thread):
> >
> >
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201604.mbox/browser
> >
> > Pasting here for ease:
> >
> > On an unrelated note, offline I was pitching to a bunch of
> > contributors another idea to deal
> > with rotting trunk post 3.x: *Make 3.x releases off of trunk directly*.
> >
> > What this gains us is that
> >  - Trunk is always nearly stable or nearly ready for releases
> >  - We no longer have some code lying around in some branch (today’s
> > trunk) that is not releasable
> > because it gets mixed with other undesirable and incompatible changes.
> >  - This needs to be coupled with more discipline on individual
> > features - medium to to large
> > features are always worked upon in branches and get merged into trunk
> > (and a nearing release!)
> > when they are ready
> >  - All incompatible changes go into some sort of a trunk-incompat
> > branch and stay there till
> > we accumulate enough of those to warrant another major release.
> >
> > Regarding "trunk-incompat", since we're still in the alpha stage for
> 3.0.0,
> > there's no need for this branch yet. This aspect of Vinod's proposal was
> > still under a bit of discussion; Chris Douglas though we should cut a
> > branch-3 for the first 3.0.0 beta, which aligns with my original
> 

Re: CHANGES.txt is gone from trunk, branch-2, branch-2.8

[Zhe Zhang]

Great change! Saving ~1min for each commit. Thanks Andrew and Allen.

On Thu, Mar 3, 2016 at 9:32 PM Yongjun Zhang <yzh...@cloudera.com> wrote:

> That's nice, thanks Andrew and Allen.
>
> --Yongjun
>
> On Thu, Mar 3, 2016 at 9:11 PM, Andrew Wang <andrew.w...@cloudera.com>
> wrote:
>
> > Hi all,
> >
> > With the inclusion of HADOOP-12651 going back to branch-2.8, CHANGES.txt
> > and release notes are now generated by Yetus. I've gone ahead and deleted
> > the manually updated CHANGES.txt from trunk, branch-2, and branch-2.8
> > (HADOOP-11792). Many thanks to Allen for the releasedocmaker.py rewrite,
> > and the Yetus integration.
> >
> > I'll go ahead and update the HowToCommit and HowToRelease wiki pages, but
> > at a high-level, this means we no longer need to edit CHANGES.txt on new
> > commit, streamlining our commit process. CHANGES.txt updates will still
> be
> > necessary for backports to older release lines like 2.6.x and 2.7.x.
> >
> > Happy committing!
> >
> > Best,
> > Andrew
> >
>
-- 
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap