Re: [DISCUSS] A unified and open Hadoop community sync up schedule?

[Sunil Govindan]

Hi Eric

This community meeting will be starting from next week onwards

- Sunil

On Wed, Jun 19, 2019 at 9:14 PM Eric Badger
 wrote:

> Is there a YARN call today (in 16 minutes)? I saw it on the calendar until
> a few minutes ago.
>
> Eric
>
> On Tue, Jun 18, 2019 at 11:18 PM Wangda Tan  wrote:
>
> > Thanks @Wei-Chiu Chuang  . updated gdoc
> >
> > On Tue, Jun 18, 2019 at 7:35 PM Wei-Chiu Chuang 
> > wrote:
> >
> > > Thanks Wangda,
> > >
> > > I just like to make a correction -- the .ics calendar file says the
> first
> > > Wednesday for HDFS/cloud connector is in Mandarin whereas on the gdoc
> is
> > to
> > > host it on the third Wednesday.
> > >
> > > On Tue, Jun 18, 2019 at 5:29 PM Wangda Tan 
> wrote:
> > >
> > > > Hi Folks,
> > > >
> > > > I just updated doc:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
> > > > with
> > > > dial-in information, notes, etc.
> > > >
> > > > Here's a calendar to subscribe:
> > > >
> > > >
> > >
> >
> https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics
> > > >
> > > > I'm thinking to give it a try from next week, any suggestions?
> > > >
> > > > Thanks,
> > > > Wangda
> > > >
> > > > On Fri, Jun 14, 2019 at 4:02 PM Wangda Tan 
> > wrote:
> > > >
> > > > > And please let me know if you can help with coordinate logistics
> > stuff,
> > > > > cross-checking, etc. Let's spend some time next week to get it
> > > finalized.
> > > > >
> > > > > Thanks,
> > > > > Wangda
> > > > >
> > > > > On Fri, Jun 14, 2019 at 4:00 PM Wangda Tan 
> > > wrote:
> > > > >
> > > > >> Hi Folks,
> > > > >>
> > > > >> Yufei: Agree with all your opinions.
> > > > >>
> > > > >> Anu: it might be more efficient to use Google doc to track meeting
> > > > >> minutes and we can put them together.
> > > > >>
> > > > >> I just put the proposal to
> > > > >>
> > > >
> > >
> >
> https://calendar.google.com/calendar/b/3?cid=aGFkb29wLmNvbW11bml0eS5zeW5jLnVwQGdtYWlsLmNvbQ
> > > > ,
> > > > >> you can check if the proposal time works or not. If you agree, we
> > can
> > > go
> > > > >> ahead to add meeting link, google doc, etc.
> > > > >>
> > > > >> If you want to have edit permissions, please drop a private email
> to
> > > me
> > > > >> so I will add you.
> > > > >>
> > > > >> We still need more hosts, in each track, ideally we should have at
> > > least
> > > > >> 3 hosts per track just like HDFS blocks :), please volunteer, so
> we
> > > can
> > > > >> have enough members to run the meeting.
> > > > >>
> > > > >> Let's shoot by end of the next week, let's get all logistics done
> > and
> > > > >> starting community sync up series from the week of Jun 25th.
> > > > >>
> > > > >> Thanks,
> > > > >> Wangda
> > > > >>
> > > > >> Thanks,
> > > > >> Wangda
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Tue, Jun 11, 2019 at 10:23 AM Anu Engineer <
> > aengin...@cloudera.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >>> For Ozone, we have started using the Wiki itself as the agenda
> and
> > > > after
> > > > >>> the meeting is over, we convert it into the meeting notes.
> > > > >>> Here is an example, the project owner can edit and maintain it,
> it
> > is
> > > > >>> like 10 mins work - and allows anyone to add stuff into the
> agenda
> > > too.
> > > > >>>
> > > > >>>
> > > > >>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/HADOOP/2019-06-10+Meeting+notes
> > > > >>>
> > > > >>> --Anu
> > > > >>>
> > > > >>> On Tue, Jun 11, 2019 at 10:20 AM Yufei Gu 
> > > > wrote:
> > > > >>>
> > > >  +1 for this idea. Thanks Wangda for bringing this up.
> > > > 
> > > >  Some comments to share:
> > > > 
> > > > - Agenda needed to be posted ahead of meeting and welcome any
> > > >  interested
> > > > party to contribute to topics.
> > > > - We should encourage more people to attend. That's whole
> point
> > > of
> > > >  the
> > > > meeting.
> > > > - Hopefully, this can mitigate the situation that some
> patches
> > > are
> > > > waiting for review for ever, which turns away new
> contributors.
> > > > - 30m per session sounds a little bit short, we can try it
> out
> > > and
> > > >  see
> > > > if extension is needed.
> > > > 
> > > >  Best,
> > > > 
> > > >  Yufei
> > > > 
> > > >  `This is not a contribution`
> > > > 
> > > > 
> > > >  On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan 
> > > > wrote:
> > > > 
> > > >  > Hi Hadoop-devs,
> > > >  >
> > > >  > Previous we have regular YARN community sync up (1 hr,
> biweekly,
> > > but
> > > >  not
> > > >  > open to public). Recently because of changes in our schedules,
> > > Less
> > > >  folks
> > > >  > showed up in the sync up for the last several months.
> > > >  >
> > > >  > I saw the K8s community did a pretty good job to run their sig
> > > >  meetings,
> > > >  > there's

Re: [VOTE] Force "squash and merge" option for PR merge on github UI

[Sunil Govindan]

Hi All

INFRA-18777 is closed and github UI has disabled #1 and #3. Only Squash and
Merge is possible.
Could we start using this option (merge from UI) from now onwards ?

- Sunil

On Mon, Jul 22, 2019 at 10:24 AM Bharat Viswanadham
 wrote:

> +1 for squash and merge.
>
> And if we use Github UI, the original author will be shown as the original
> author of the code, not who clicks the squash and merge.
>
> [image: Screen Shot 2019-07-17 at 11.58.51 AM.png]
>
> Like in the screenshot arp7 committed the change from GithubUI, but the
> author is still been shown as the original author "bharatviswa504".
>
> * ef66e4999f3 N - HDDS-1666. Issue in openKey when allocating block.
> (#943) (2 days ago)  
> Thanks,
> Bharat
>
>
>
> On Wed, Jul 17, 2019 at 10:20 AM Iñigo Goiri  wrote:
>
>> +1
>>
>> On Wed, Jul 17, 2019 at 4:17 AM Steve Loughran
>> 
>> wrote:
>>
>> > +1 for squash and merge, with whoever does the merge adding the full
>> commit
>> > message for the logs, with JIRA, contributor(s) etc
>> >
>> > One limit of the github process is that the author of the commit becomes
>> > whoever hit the squash button, not whoever did the code, so it loses the
>> > credit they are due. This is why I'm doing local merges (With some help
>> > from smart-apply-patch). I think I'll have to explore smart-apply-patch
>> to
>> > see if I can do even more with it
>> >
>> >
>> >
>> >
>> > On Wed, Jul 17, 2019 at 7:07 AM Elek, Marton  wrote:
>> >
>> > > Hi,
>> > >
>> > > Github UI (ui!) helps to merge Pull Requests to the proposed branch.
>> > > There are three different ways to do it [1]:
>> > >
>> > > 1. Keep all the different commits from the PR branch and create one
>> > > additional merge commit ("Create a merge commit")
>> > >
>> > > 2. Squash all the commits and commit the change as one patch ("Squash
>> > > and merge")
>> > >
>> > > 3. Keep all the different commits from the PR branch but rebase, merge
>> > > commit will be missing ("Rebase and merge")
>> > >
>> > >
>> > >
>> > > As only the option 2 is compatible with the existing development
>> > > practices of Hadoop (1 issue = 1 patch = 1 commit), I call for a lazy
>> > > consensus vote: If no objections withing 3 days, I will ask INFRA to
>> > > disable the options 1 and 3 to make the process less error prone.
>> > >
>> > > Please let me know, what do you think,
>> > >
>> > > Thanks a lot
>> > > Marton
>> > >
>> > > ps: Personally I prefer to merge from local as it enables to sign the
>> > > commits and do a final build before push. But this is a different
>> story,
>> > > this proposal is only about removing the options which are obviously
>> > > risky...
>> > >
>> > > ps2: You can always do any kind of merge / commits from CLI, for
>> example
>> > > to merge a feature branch together with keeping the history.
>> > >
>> > > [1]:
>> > >
>> > >
>> >
>> https://help.github.com/en/articles/merging-a-pull-request#merging-a-pull-request-on-github
>> > >
>> > > -
>> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> > >
>> > >
>> >
>>
>

Re: Thoughts about moving submarine to a separate git repo?

[Sunil Govindan]

Hi Xun,

Thanks for the proposal. Altogether this proposal makes sense to me.

IIUC, an external repo is created and developers found more productive in
that because of faster reviews and commits.
To do more this apache way,
1. I think its better we track all this in apache jira (apparently this is
already happening, thanks for that).
2. In apache, we have a provision called branch committer. This role will
help to drive a feature ('s) in a branch cut from apache trunk,
and selected branch committers can push changes to apache repo. Thus fast
pacing feature development in community.
So nominating couple of active contributors of submarine for branch
committership could address the concerns raised by you above.

For this part, only difference I see is about a new repo instead of branch
from Apache trunk.
AFAIK, This seems not a stopper for the branch committership suggestion
(please correct me if I am wrong).
However we need to ensure that commit rights of that repo is only available
branch committer's and all apache policies are adhered in that.

Overall I feel option #2 mentioned above is a good way to proceed further.
Welcoming thoughts from others who have more contexts into similar past
experiences.

Thanks,
Sunil


On Fri, Aug 16, 2019 at 10:13 AM Xun Liu  wrote:

> Dear Submarine developers,
>
> My name is Xun Liu, I am a member of the Hadoop submarine development team.
> I'm one of the major contributor of Submarine since June 2018.
>
> I want to hear your thoughts about creating a separate GitHub repo under
> Apache to do submarine development. This is an independent effort of
> Submarine spin-off from the Hadoop project [
>
> https://lists.apache.org/thread.html/3fab657f905d081b536d9081dc404f7fd20c80eb824c857bc8e16e3b@
> ].
> However, once the spin-off is approved, this effort can benefit the
> follow-up processes as well.
>
> Submarine dev community has a total of 8 developers and submits an average
> of 4 to 5 PR per day.
> But there are a limited number of Hadoop committer actively help review and
> merge patches, which causes development progress delays.
>
> So we created an external GitHub repo [
> https://github.com/hadoopsubmarine/submarine] and moved all the code for
> the Hadoop submarine project into the external Github repo.
> In this way, everyone can review the code for each other, and now the
> development progress of Hadoop submarine is very fast.
>
> Also, now Submarine has little dependency on Hadoop, we want to have a
> separate CI/CD pipeline to release and test submarine instead of every time
> build whole Hadoop. Putting Submarine under Hadoop will introduce
> unnecessary dependencies to Hadoop's top-level pom.xml.
>
> Our development process still complies with the development rules of the
> Hadoop community: first, create a ticket in the submarine JIRA, and then
> develop, in the external GitHub repo repository, the title of each PR will
> be accompanied by the JIRA ID number.
>
> Once the Apache Github repo is created, we going to move all external
> commits to the new Apache Github repo.
>
> Any suggestions are welcome!
>
> Best Regards
> Xun Liu
>

Re: [VOTE] Mark 2.6, 2.7, 3.0 release lines EOL

[Sunil Govindan]

+1

- Sunil

On Wed, Aug 21, 2019 at 8:33 AM Wangda Tan  wrote:

> Hi all,
>
> This is a vote thread to mark any versions smaller than 2.7 (inclusive),
> and 3.0 EOL. This is based on discussions of [1]
>
> This discussion runs for 7 days and will conclude on Aug 28 Wed.
>
> Please feel free to share your thoughts.
>
> Thanks,
> Wangda
>
> [1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201908.mbox/%3cCAD++eC=ou-tit1faob-dbecqe6ht7ede7t1dyra2p1yinpe...@mail.gmail.com%3e
> ,
>

Re: Thoughts about moving submarine to a separate git repo?

[Sunil Govindan]

Thanks Wangda for sharing the thoughts.

I agree with the idea of new repo for more cleaner code base and remove
additional dependencies, jenkins etc.
One major point from my end is that the commits to new repo ideally should
happen from committers or branch committers.

As hadoop community, we could help in this case. @Wangda Tan
  your thoughts?

Thanks
Sunil

On Wed, Aug 21, 2019 at 9:47 AM Wangda Tan  wrote:

> Hi Xun,
>
> Thanks for starting this thread. I'm glad to see the existing momentum made
> by Submarine community, and I like the proposal to make it be a separate
> Git repo.
>
> Some suggestions:
>
> 1) For options mentioned by Sunil: I think it's better to be a separate Git
> repo instead of a branch. To me, the branch is targeted for a
> diverged codebase instead of a new codebase. Since Submarine needs a clean
> root source code directory. I think moving to a new Git repo makes more
> sense.
>
> 2) For the Submarine external code, when we pulling them in, I think we
> need to make sure license, iCLA, code comply with Apache standard. Which
> means we need to do some additional reviews, etc. for patches being pulled
> in. (instead of as-is).
>
> 3) Can we address comments from @Wei-Chiu Chuang  , to
> give some extra time for existing Hadoop committers/contributors who have
> interests to review the code? Waiting for at least 1 day for a big patch
> and 6 hours for a minor fix might be a good rule to follow. And @Wei-Chiu
> Chuang  please let Submarine community know if you
> have
> anything interested to review so developers can ping you when they have any
> patches.
>
> Best,
> Wangda
>
> On Tue, Aug 20, 2019 at 9:54 PM Wei-Chiu Chuang 
> wrote:
>
> > >
> > >
> > >
> > > Submarine dev community has a total of 8 developers and submits an
> > average
> > > of 4 to 5 PR per day.
> > > But there are a limited number of Hadoop committer actively help review
> > and
> > > merge patches, which causes development progress delays.
> > >
> > > I just want to point this out that this is concerning -- I wanted to
> help
> > review patches, but it wasn't obvious the patches were raised as PRs in a
> > non-apache git repo.
> > Please be more inclusive.
> >
>

Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Sunil Govindan]

+1

- Sunil

On Sat, Aug 24, 2019 at 7:36 AM Wangda Tan  wrote:

> Hi devs,
>
> This is a voting thread to move Submarine source code, documentation from
> Hadoop repo to a separate Apache Git repo. Which is based on discussions of
>
> https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E
>
> Contributors who have permissions to push to Hadoop Git repository will
> have permissions to push to the new Submarine repository.
>
> This voting thread will run for 7 days and will end at Aug 30th.
>
> Please let me know if you have any questions.
>
> Thanks,
> Wangda Tan
>

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Sunil Govindan]

Thanks Vinay for starting the thread.

I agree to Anu's view point related to protobuf. And with the suggestion
pointed out by Duo Zhang, if we can make use
of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
of protobuf will also be more easier.

However i think its better to do this effort in trunk itself.
In offline talks, few members were interested to start 3.3.0 release. And
given that happens soon, I feel its better
we do this task in trunk itself as branch diverge is very much possible.
And to bring to call a merge on such a big branch will be even more tough
task.

my 2 cents.

Thanks
Sunil

On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang)  wrote:

> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate
> the protobuf code. It will download the protoc binary from the maven
> central so we do not need to install protoc on the build machine any more.
>
> Zhenyu Zheng  于2019年9月4日周三 下午5:27写道：
>
> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
> for
> > over 2 month related to the Protobuf problem .
> > According to the latest successful build log:
> >
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> > the
> > os version was ubuntu 14.04 and for the jobs that are failling now such
> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> > the os version is 18.04. I'm not very familiar with the version changing
> > for the jobs but I did a little search, according to:
> >
> >
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> > &
> >
> >
> https://packages.ubuntu.com/search?suite=default§ion=all&arch=any&keywords=libprotoc-dev&searchon=names
> > it both said that the version of libprotc-dev and protobuf-compiler
> > available for ubuntu 18.04 is 3.0.0
> >
> >
> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena  wrote:
> >
> >> Thanx Vinay for the initiative, Makes sense to add support for different
> >> architectures.
> >>
> >> +1, for the branch idea.
> >> Good Luck!!!
> >>
> >> -Ayush
> >>
> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) 
> >> wrote:
> >> >
> >> > For HBase, we purged all the protobuf related things from the public
> >> API,
> >> > and then upgraded to a shaded and relocated version of protobuf. We
> have
> >> > created a repo for this:
> >> >
> >> > https://github.com/apache/hbase-thirdparty
> >> >
> >> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
> >> our
> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
> >> discuss
> >> > on how to deal with the upgrading of coprocessor. Glad to see that the
> >> > hadoop community is also willing to solve the problem.
> >> >
> >> > Anu Engineer  于2019年9月3日周二 上午1:23写道：
> >> >
> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
> that
> >> >> Hadoop and the downstream projects work correctly after you upgrade
> >> core
> >> >> components like Protobuf.
> >> >> So while branching and working on a branch is easy, merging back
> after
> >> you
> >> >> upgrade some of these core components is insanely hard. You might
> want
> >> to
> >> >> make sure that community buys into upgrading these components in the
> >> trunk.
> >> >> That way we will get testing and downstream components will notice
> when
> >> >> things break.
> >> >>
> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
> long
> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
> >> that
> >> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
> >> base.
> >> >> It has been rightly pointed to me that while all the arguments I make
> >> is
> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
> >> worst
> >> >> part is we will not even know what breaks until downstream projects
> >> pick up
> >> >> these changes and work against us.
> >> >>
> >> >> If we work off the Hadoop version 3 — and assume that we have
> >> "shading" in
> >> >> place for all deployments; it might be possible to get there; still a
> >> >> daunting task.
> >> >>
> >> >> So best of luck with the branch approach — But please remember,
> Merging
> >> >> back will be hard, Just my 2 cents.
> >> >>
> >> >> — Anu
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
> zhengzhenyul...@gmail.com
> >> >
> >> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
> >> separate
> >> >>> branch with it's own ARM CI seems a really good idea.
> >> >>> By doing this we won't break any of the undergoing development in
> >> trunk
> >> >> and
> >> >>> a CI can be a very good way to show what are the
> >> >>> current problems and what have been fixed, it will also provide a
> very
> >> >> good
> >> >>> view for contributors that are intrested to working on
> >> >>> this. We can finally merge back the branch to trunk until the
> >> community
> >>

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Sunil Govindan]

+1 to the proposal.
Thanks to the community for the great response.

- Sunil

On Sun, Sep 1, 2019 at 10:49 AM Wangda Tan  wrote:

> Hi all,
>
> As we discussed in the previous thread [1],
>
> I just moved the spin-off proposal to CWIKI and completed all TODO parts.
>
>
> https://cwiki.apache.org/confluence/display/HADOOP/Submarine+Project+Spin-Off+to+TLP+Proposal
>
> If you have interests to learn more about this. Please review the proposal
> let me know if you have any questions/suggestions for the proposal. This
> will be sent to board post voting passed. (And please note that the
> previous voting thread [2] to move Submarine to a separate Github repo is a
> necessary effort to move Submarine to a separate Apache project but not
> sufficient so I sent two separate voting thread.)
>
> Please let me know if I missed anyone in the proposal, and reply if you'd
> like to be included in the project.
>
> This voting runs for 7 days and will be concluded at Sep 7th, 11 PM PDT.
>
> Thanks,
> Wangda Tan
>
> [1]
>
> https://lists.apache.org/thread.html/4a2210d567cbc05af92c12aa6283fd09b857ce209d537986ed800029@%3Cyarn-dev.hadoop.apache.org%3E
> [2]
>
> https://lists.apache.org/thread.html/6e94469ca105d5a15dc63903a541bd21c7ef70b8bcff475a16b5ed73@%3Cyarn-dev.hadoop.apache.org%3E
>

Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0

[Sunil Govindan]

Hi Rohith

Thanks for putting this together, appreciate the same.

+1 (binding)

- verified signature
- brought up a cluster from the tar ball
- Ran some basic MR jobs
- RM UI seems fine (old and new)


Thanks
Sunil

On Wed, Sep 11, 2019 at 12:56 PM Rohith Sharma K S <
rohithsharm...@apache.org> wrote:

> Hi folks,
>
> I have put together a release candidate (RC0) for Apache Hadoop 3.2.1.
>
> The RC is available at:
> http://home.apache.org/~rohithsharmaks/hadoop-3.2.1-RC0/
>
> The RC tag in git is release-3.2.1-RC0:
> https://github.com/apache/hadoop/tree/release-3.2.1-RC0
>
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1226/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> This vote will run for 7 days(5 weekdays), ending on 18th Sept at 11:59 pm
> PST.
>
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
>
> Thanks & Regards
> Rohith Sharma K S
>

Re: [VOTE] Release Hadoop-3.1.3-RC0

[Sunil Govindan]

+1 (binding)

Thanks Zhankun for putting up the release. Thanks for leading this.

- verified signature
- ran a local cluster from tar ball
- ran some MR jobs
- perform CLI ops, and looks good
- UI seems fine

Thanks
Sunil

On Thu, Sep 12, 2019 at 1:34 PM Zhankun Tang  wrote:

> Hi folks,
>
> Thanks to everyone's help on this release. Special thanks to Rohith,
> Wei-Chiu, Akira, Sunil, Wangda!
>
> I have created a release candidate (RC0) for Apache Hadoop 3.1.3.
>
> The RC release artifacts are available at:
> http://home.apache.org/~ztang/hadoop-3.1.3-RC0/
>
> The maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1228/
>
> The RC tag in git is here:
> https://github.com/apache/hadoop/tree/release-3.1.3-RC0
>
> And my public key is at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> *This vote will run for 7 days, ending on Sept.19th at 11:59 pm PST.*
>
> For the testing, I have run several Spark and distributed shell jobs in my
> pseudo cluster.
>
> My +1 (non-binding) to start.
>
> BR,
> Zhankun
>
> On Wed, 4 Sep 2019 at 15:56, zhankun tang  wrote:
>
> > Hi all,
> >
> > Thanks for everyone helping in resolving all the blockers targeting
> Hadoop
> > 3.1.3[1]. We've cleaned all the blockers and moved out non-blockers
> issues
> > to 3.1.4.
> >
> > I'll cut the branch today and call a release vote soon. Thanks!
> >
> >
> > [1]. https://s.apache.org/5hj5i
> >
> > BR,
> > Zhankun
> >
> >
> > On Wed, 21 Aug 2019 at 12:38, Zhankun Tang  wrote:
> >
> >> Hi folks,
> >>
> >> We have Apache Hadoop 3.1.2 released on Feb 2019.
> >>
> >> It's been more than 6 months passed and there're
> >>
> >> 246 fixes[1]. 2 blocker and 4 critical Issues [2]
> >>
> >> (As Wei-Chiu Chuang mentioned, HDFS-13596 will be another blocker)
> >>
> >>
> >> I propose my plan to do a maintenance release of 3.1.3 in the next few
> >> (one or two) weeks.
> >>
> >> Hadoop 3.1.3 release plan:
> >>
> >> Code Freezing Date: *25th August 2019 PDT*
> >>
> >> Release Date: *31th August 2019 PDT*
> >>
> >>
> >> Please feel free to share your insights on this. Thanks!
> >>
> >>
> >> [1] https://s.apache.org/zw8l5
> >>
> >> [2] https://s.apache.org/fjol5
> >>
> >>
> >> BR,
> >>
> >> Zhankun
> >>
> >
>

[DISCUSS] Release Docs pointers Hadoop site

[Sunil Govindan]

Hi Folks,

At present,
http://hadoop.apache.org/docs/stable/  points to *Apache Hadoop 3.2.1*
http://hadoop.apache.org/docs/current/ points to *Apache Hadoop 3.2.1*
http://hadoop.apache.org/docs/stable2/  points to *Apache Hadoop 2.9.2*
http://hadoop.apache.org/docs/current2/ points to *Apache Hadoop 2.9.2*

3.2.1 is released last day. *Now 3.1.3 has completed voting* and it is in
the final stages of staging
As per me,
a) 3.2.1 will be still be pointing to http://hadoop.apache.org/docs/stable/
?
b) 3.1.3 should be pointing to http://hadoop.apache.org/docs/current/ ?

Now my questions,
1. But if the release manager of 3.1 line thinks 3.1.3 is stable, and 3.2
line is also in stable state, which release should get precedence to be
called as *stable* in any release line (2.x or 3.x) ?
or do we need a vote or discuss thread to decide which release shall be
called as stable per release line?
2. Given 3.2.1 is released and pointing to 3.2.1 as stable, then when 3.1.3
is getting released now, could http://hadoop.apache.org/docs/current/ shall
be updated to 3.1.3 ? is it the norms ?

Thanks
Sunil

Re: [DISCUSS] Release Docs pointers Hadoop site

[Sunil Govindan]

Bumping up this thread again for feedback.
@Zhankun Tang   is now waiting for a confirmation to
complete 3.1.3 release publish activities.

- Sunil

On Fri, Sep 27, 2019 at 11:03 AM Sunil Govindan  wrote:

> Hi Folks,
>
> At present,
> http://hadoop.apache.org/docs/stable/  points to *Apache Hadoop 3.2.1*
> http://hadoop.apache.org/docs/current/ points to *Apache Hadoop 3.2.1*
> http://hadoop.apache.org/docs/stable2/  points to *Apache Hadoop 2.9.2*
> http://hadoop.apache.org/docs/current2/ points to *Apache Hadoop 2.9.2*
>
> 3.2.1 is released last day. *Now 3.1.3 has completed voting* and it is in
> the final stages of staging
> As per me,
> a) 3.2.1 will be still be pointing to
> http://hadoop.apache.org/docs/stable/ ?
> b) 3.1.3 should be pointing to http://hadoop.apache.org/docs/current/ ?
>
> Now my questions,
> 1. But if the release manager of 3.1 line thinks 3.1.3 is stable, and 3.2
> line is also in stable state, which release should get precedence to be
> called as *stable* in any release line (2.x or 3.x) ?
> or do we need a vote or discuss thread to decide which release shall be
> called as stable per release line?
> 2. Given 3.2.1 is released and pointing to 3.2.1 as stable, then when
> 3.1.3 is getting released now, could
> http://hadoop.apache.org/docs/current/ shall be updated to 3.1.3 ? is it
> the norms ?
>
> Thanks
> Sunil
>

Re: [DISCUSS] Remove Ozone and Submarine from Hadoop repo

[Sunil Govindan]

+1 on this to remove staleness.

- Sunil

On Thu, Oct 24, 2019 at 12:51 PM Akira Ajisaka  wrote:

> Hi folks,
>
> Both Ozone and Apache Submarine have separate repositories.
> Can we remove these modules from hadoop-trunk?
>
> Regards,
> Akira
>

[DISCUSS] hadoop-kafka module not maintained to latest version

[Sunil Govindan]

Hi Folks,

hadoop-kafka module seems using a very old version of kafka (0.8.2.1). This
causes a lot of conflicts while using together for a build.

Few questions/proposals here:
1. If there are no usage for this old version of kafka module, could this
module be removed from hadoop repo?
2. Given this module is needed by certain use case scenarios, how to make
it's been maintained regularly?

Please help to share some thoughts on this.

Thanks
Sunil

Re: [DISCUSS] EOL Hadoop branch-2.8

[Sunil Govindan]

Hi Wei-Chiu

Extremely sorry for the late reply here.
Cud u pls help to add more clarity on defining what will happen for
branch-2.8 when we call EOL.
Does this mean that, no more release coming out from this branch, or some
more additional guidelines?

- Sunil


On Mon, Feb 24, 2020 at 11:47 PM Wei-Chiu Chuang
 wrote:

> This thread has been running for 7 days and no -1.
>
> Don't think we've established a formal EOL process, but to publicize the
> EOL, I am going to file a jira, update the wiki and post the announcement
> to general@ and user@
>
> On Wed, Feb 19, 2020 at 1:40 PM Dinesh Chitlangia 
> wrote:
>
> > Thanks Wei-Chiu for initiating this.
> >
> > +1 for 2.8 EOL.
> >
> > On Tue, Feb 18, 2020 at 10:48 PM Akira Ajisaka 
> > wrote:
> >
> > > Thanks Wei-Chiu for starting the discussion,
> > >
> > > +1 for the EoL.
> > >
> > > -Akira
> > >
> > > On Tue, Feb 18, 2020 at 4:59 PM Ayush Saxena 
> wrote:
> > >
> > > > Thanx Wei-Chiu for initiating this
> > > > +1 for marking 2.8 EOL
> > > >
> > > > -Ayush
> > > >
> > > > > On 17-Feb-2020, at 11:14 PM, Wei-Chiu Chuang 
> > > wrote:
> > > > >
> > > > > The last Hadoop 2.8.x release, 2.8.5, was GA on September 15th
> 2018.
> > > > >
> > > > > It's been 17 months since the release and the community by and
> large
> > > have
> > > > > moved up to 2.9/2.10/3.x.
> > > > >
> > > > > With Hadoop 3.3.0 over the horizon, is it time to start the EOL
> > > > discussion
> > > > > and reduce the number of active branches?
> > > >
> > > > -
> > > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > > >
> > > >
> > >
> >
>

JENKINS is not posting build results in jiras

[Sunil Govindan]

Hello

>From last few days, jenkins results are not getting published in jiras.

For eg, https://builds.apache.org/job/PreCommit-YARN-Build/25735/console has
results. but I cannot see the same in YARN-9879. Similarly in the cases
of YARN-10198 etc.

I could see a message in console o/p as "Adding comment to JIRA". But its
not happening.

Any ideas?

Thanks
Sunil

Re: JENKINS is not posting build results in jiras

[Sunil Govindan]

Thanks Akira for helping.

Best Regards,
Sunil

On Wed, Mar 25, 2020 at 6:41 AM Akira Ajisaka  wrote:

> Sorry it was the root cause. Reverted.
> Now the precommit job can add comments to GitHub.
> https://github.com/apache/hadoop/pull/1820#issuecomment-603545202
>
> Akira
>
> On Tue, Mar 24, 2020 at 8:53 PM Akira Ajisaka  wrote:
>
> > A few days ago I upgraded the build image and maybe it is the root cause.
> > Created a pull request to revert the change to see if it is really the
> > cause: https://github.com/apache/hadoop/pull/1911
> >
> > Thanks,
> > Akira
> >
> > On Tue, Mar 24, 2020 at 8:21 PM Akira Ajisaka 
> wrote:
> >
> >> Thanks Sunil for the report.
> >>
> >> The precommit jobs are failing to add comments to GitHub also.
> >> I'm looking into the cause, and any help is appreciated.
> >>
> >> Thanks,
> >> Akira
> >>
> >>
> >> On Mon, Mar 23, 2020 at 7:39 PM Sunil Govindan 
> wrote:
> >>
> >>> Hello
> >>>
> >>> From last few days, jenkins results are not getting published in jiras.
> >>>
> >>> For eg,
> https://builds.apache.org/job/PreCommit-YARN-Build/25735/console
> >>> has
> >>> results. but I cannot see the same in YARN-9879. Similarly in the cases
> >>> of YARN-10198 etc.
> >>>
> >>> I could see a message in console o/p as "Adding comment to JIRA". But
> its
> >>> not happening.
> >>>
> >>> Any ideas?
> >>>
> >>> Thanks
> >>> Sunil
> >>>
> >>
>

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

[Sunil Govindan]

Yes, lets wait till weekend as I can see progress in these blockers.

Thanks
Sunil

On Tue, Apr 21, 2020 at 11:35 AM Brahma Reddy Battula 
wrote:

> Sure, I will do that.
>
> Since blockers are not closed, I didn't cut the branch because
> multiple branches might confuse or sombody might miss to commit.Shall I
> wait till this weekend to create..?
>
> On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka 
> wrote:
>
> > Hi Brahma,
> >
> > Thank you for preparing the release.
> > Could you cut branch-3.3.0? I would like to backport some fixes for 3.3.1
> > and not for 3.3.0.
> >
> > Thanks and regards,
> > Akira
> >
> > On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula  >
> > wrote:
> >
> >> Hi All,
> >>
> >> we are down to two blockers issues now (YARN-10194 and YARN-9848) which
> >> are in patch available state.Hopefully we can out the RC soon.
> >>
> >> thanks to @Prabhu Joseph  ,@masakate,@akira
> >> and @Wei-Chiu Chuang   and others for helping
> >> resloving the blockers.
> >>
> >>
> >>
> >> On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula <
> bra...@apache.org>
> >> wrote:
> >>
> >>>
> >>> @Prabhu Joseph 
> >>> >>> Have committed the YARN blocker YARN-10219 to trunk and
> >>> cherry-picked to branch-3.3. Right now, there are two blocker Jiras -
> >>> YARN-10233 and HADOOP-16982
> >>> which i will help to review and commit. Thanks.
> >>>
> >>> Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as a
> >>> blockers. (without YARN-10233 we have given so many releases,it's not
> newly
> >>> introduced.).. Thanks
> >>>
> >>> @Vinod Kumar Vavilapalli  ,@adam Antal,
> >>>
> >>> I noted YARN-9848 as a blocker as you mentioned above.
> >>>
> >>> @All,
> >>>
> >>> Currently following four blockers are pending for 3.3.0 RC.
> >>>
> >>> HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848.
> >>>
> >>>
> >>>
> >>> On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli <
> >>> vino...@apache.org> wrote:
> >>>
>  Looks like a really bad bug to me.
> 
>  +1 for revert and +1 for making that a 3.3.0 blocker. I think should
>  also revert it in a 3.2 maintenance release too.
> 
>  Thanks
>  +Vinod
> 
>  > On Apr 14, 2020, at 5:03 PM, Adam Antal  .INVALID>
>  wrote:
>  >
>  > Hi everyone,
>  >
>  > Sorry for coming a bit late with this, but there's also one jira
> that
>  can
>  > have potential impact on clusters and we should talk about it.
>  >
>  > Steven Rand found this problem earlier and commented to
>  > https://issues.apache.org/jira/browse/YARN-4946.
>  > The bug has impact on the RM state store: the RM does not delete
> apps
>  - see
>  > more details in his comment here:
>  >
> 
> https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599
>  > .
>  > (FYI He also created
> https://issues.apache.org/jira/browse/YARN-9848
>  with
>  > the revert task).
>  >
>  > It might not be an actual blocker, but since there wasn't any
>  consensus
>  > about a follow up action, I thought we should decide how to proceed
>  before
>  > release 3.3.0.
>  >
>  > Regards,
>  > Adam
>  >
>  > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph <
>  prabhujose.ga...@gmail.com>
>  > wrote:
>  >
>  >> Thanks Brahma for the update.
>  >>
>  >> Have committed the YARN blocker YARN-10219 to trunk and
>  cherry-picked to
>  >> branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and
>  >> HADOOP-16982
>  >> which i will help to review and commit. Thanks.
>  >>
>  >> [image: Screen Shot 2020-04-14 at 1.01.51 PM.png]
>  >>
>  >> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> (Blocker,
>  >> Critical) AND resolution = Unresolved AND "Target Version/s" =
> 3.3.0
>  ORDER
>  >> BY priority DESC
>  >>
>  >>
>  >> On Sun, Apr 12, 2020 at 12:19 AM Brahma Reddy Battula <
>  bra...@apache.org>
>  >> wrote:
>  >>
>  >>> *Pending for 3.3.0 Release:*
>  >>>
>  >>> One Blocker(HADOOP-16963) confirmation and following jira's are
>  open as
>  >>> these needs to merged to other branches(I am tracking the same,
>  Ideally
>  >>> this can be closed and can raise seperate jira's to track).
>  >>>
>  >>>
>  >>> 1–4 of 4Refresh results
>  >>> <
>  >>>
> 
> https://issues.apache.org/jira/issues/?jql=project%20in%20(%22Hadoop%20HDFS%22)%20AND%20resolution%20%3D%20Unresolved%20AND%20(cf%5B12310320%5D%20%3D%203.3.0%20OR%20fixVersion%20%3D%203.3.0)%20ORDER%20BY%20priority%20DESC#
>  
>  >>> Columns
>  >>> Patch InfoKeyTSummaryAssigneeReporterP
>  StatusResolutionUpdatedDueCreated
>  >>>  HDFS-14353
>  >>> 

Re: [DISCUSS] making Ozone a separate Apache project

[Sunil Govindan]

+1

Thanks
Sunil

On Wed, May 13, 2020 at 1:23 PM Elek, Marton  wrote:

>
>
> I would like to start a discussion to make a separate Apache project for
> Ozone
>
>
>
> ### HISTORY [1]
>
>   * Apache Hadoop Ozone development started on a feature branch of
> Hadoop repository (HDFS-7240)
>
>   * In the October of 2017 a discussion has been started to merge it to
> the Hadoop main branch
>
>   * After a long discussion it's merged to Hadoop trunk at the March of
> 2018
>
>   * During the discussion of the merge, it was suggested multiple times
> to create a separated project for the Ozone. But at that time:
>  1). Ozone was tightly integrated with Hadoop/HDFS
>  2). There was an active plan to use Block layer of Ozone (HDDS or
> HDSL at that time) as the block level of HDFS
>  3). The community of Ozone was a subset of the HDFS community
>
>   * The first beta release of Ozone was just released. Seems to be a
> good time before the first GA to make a decision about the future.
>
>
>
> ### WHAT HAS BEEN CHANGED
>
>   During the last years Ozone became more and more independent both at
> the community and code side. The separation has been suggested again and
> again (for example by Owen [2] and Vinod [3])
>
>
>
>   From COMMUNITY point of view:
>
>
>* Fortunately more and more new contributors are helping Ozone.
> Originally the Ozone community was a subset of HDFS project. But now a
> bigger and bigger part of the community is related to Ozone only.
>
>* It seems to be easier to _build_ the community as a separated project.
>
>* A new, younger project might have different practices
> (communication, commiter criteria, development style) compared to old,
> mature project
>
>* It's easier to communicate (and improve) these standards in a
> separated projects with clean boundaries
>
>* Separated project/brand can help to increase the adoption rate and
> attract more individual contributor (AFAIK it has been seen in Submarine
> after a similar move)
>
>   * Contribution process can be communicated more easily, we can make
> first time contribution more easy
>
>
>
>   From CODE point of view Ozone became more and more independent:
>
>
>   * Ozone has different release cycle
>
>   * Code is already separated from Hadoop code base
> (apache/hadoop-ozone.git)
>
>   * It has separated CI (github actions)
>
>   * Ozone uses different (more strict) coding style (zero toleration of
> unit test / checkstyle errors)
>
>   * The code itself became more and more independent from Hadoop on
> Maven level. Originally it was compiled together with the in-tree latest
> Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC,
> Configuration...)
>
>   * It starts to use multiple version of Hadoop (on client side)
>
>   * Volume of resolved issues are already very high on Ozone side (Ozone
> had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all
> together in the last 2-3 months)
>
>
> Summary: Before the first Ozone GA release, It seems to be a good time
> to discuss the long-term future of Ozone. Managing it as a separated TLP
> project seems to have more benefits.
>
>
> Please let me know what your opinion is...
>
> Thanks a lot,
> Marton
>
>
>
>
>
> [1]: For more details, see:
> https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md
>
> [2]:
>
> https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E
>
> [3]:
>
> https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Removing the archiac master branch

[Sunil Govindan]

+1 for initiating this!

Sunil

On Fri, Jun 19, 2020 at 10:50 PM Owen O'Malley 
wrote:

> We unfortunately have a lot of master/slave and whitelist/blacklist
> terminology usage in Hadoop. It will take a while to fix them all, but one
> is easy to fix. In particular, we have a "master" branch that hasn't been
> used since the project reunification and we use "trunk" as the main branch.
>
> I propose that we delete the "master" branch. Thoughts?
>
> .. Owen
>

Re: [Virtual MEETUP]: Migration to Hadoop 3

[Sunil Govindan]

Hi folks

great to see the participation and thanks all for joining!

@Wei-Chiu Chuang  3.1 is one of the stable line along
with 3.2
As we are moving to a stable 3.3, we should start to visit the fate of 3.1
by end of this year and early next year.

Thanks
Sunil

On Thu, Aug 27, 2020 at 11:17 AM Wei-Chiu Chuang
 wrote:

> Thanks Brahma,
>
> Eric, do you have a target Hadoop 3 release line in mind?
>
> The "unofficial" plan here at Cloudera is to rebase our current dev
> codebase from Hadoop 3.1.1 to 3.3 some time later. The Hadoop 3.1 code line
> will approach its 3rd anniversary by this year's end so perhaps we can
> start to sunset it.
>
> On Wed, Aug 26, 2020 at 10:51 AM Brahma Reddy Battula 
> wrote:
>
> > One more update from me.
> >
> > We didn't face any issues with YARN, for HDFS you can have a look at the
> > following jira's.
> >
> > https://issues.apache.org/jira/browse/HDFS-13596
> > https://issues.apache.org/jira/browse/HDFS-14396
> > https://issues.apache.org/jira/browse/HDFS-14509
> >
> > Following jira is incompatible for ACL commands.Only hadoop-3 clients
> will
> > work against hadoop-3 server during the upgrade.
> >
> > https://issues.apache.org/jira/browse/HDFS-6984
> >
> >
> >
> > On Wed, Aug 26, 2020 at 11:06 PM Brahma Reddy Battula  >
> > wrote:
> >
> > >
> > > Hi Eric,
> > >
> > > check the following references for the same.
> > >
> > > 01/02/2020 Didi talked about their large scale HDFS cluster upgrade
> > > experience.
> > >
> > > Slides:
> > > https://drive.google.com/open?id=1iwJ1asalYfgnOCBuE-RfeG-NpSocjIcy
> > >
> > > Recording:
> > >
> >
> https://cloudera.zoom.us/rec/share/7MF_dLX0339OY5391xvkZP8NLrXieaa8gyZK-fYJnUkGOUUXvaUh5cl_6AVYetQl
> > >
> > > Didi studied two upgrade approaches from the community documentation:
> > > express upgrade and rolling upgrade. Rolling upgrade was selected.
> > >
> > > Yahoo Japan was trying out from hadoop-2.6 to hadop-3.2.1
> > >
> > > https://techblog.yahoo.co.jp/entry/20191206786320/
> > >
> > > On Wed, Aug 26, 2020 at 6:56 PM epa...@apache.org 
> > > wrote:
> > >
> > >> Hello. Just a reminder that today I would like to invite you all to
> > >> discuss your
> > >> experiences migrating from Hadoop 2 to Hadoop 3.
> > >>
> > >> -Eric
> > >>
> > >> On Monday, August 24, 2020, 1:58:37 PM CDT, epa...@apache.org <
> > >> epa...@apache.org> wrote:
> > >>
> > >> Hello everyone!
> > >>
> > >> We are considering migrating to Hadoop 3, and we would be very
> > interested
> > >> to
> > >> hear about your experiences. If you have migrated from Hadoop 2 to
> > Hadoop
> > >> 3
> > >> and can provide insights, please kindly consider attending the
> > following:
> > >>
> > >> Date: Wednesday, Aug 26, 2020
> > >> Time: 10:00 A.M. PDT / 12:00 P.M. CDT / 01:00 P.M. EDT / 05:00 P.M.
> GMT
> > >> Location: Zoom: https://cloudera.zoom.us/j/880548968
> > >>
> > >> Hope to see you there!
> > >>
> > >> Thank you!
> > >> Eric Payne
> > >> @ Verizon Media
> > >>
> > >> -
> > >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >>
> > >>
> > >
> > > --
> > >
> > >
> > >
> > > --Brahma Reddy Battula
> > >
> >
> >
> > --
> >
> >
> >
> > --Brahma Reddy Battula
> >
>

Re: [DISCUSS] fate of branch-2.9

[Sunil Govindan]

+1

Thanks
Sunil

On Thu, Aug 27, 2020 at 3:49 PM Xiaoqiao He  wrote:

> +1 for putting 2.9 release lines to EOL.
>
> Thanks,
> Hexiaoqiao
>
> On Thu, Aug 27, 2020 at 3:14 PM Mingliang Liu  wrote:
>
>> +1 for putting 2.9 lines to EOL.
>>
>> Let's focus on 2.10 releases for Hadoop 2. Also is there any plan for
>> 2.10.1? It has been 11 months since 2.10 first release.
>>
>> Thanks,
>>
>> On Wed, Aug 26, 2020 at 10:57 PM Wei-Chiu Chuang 
>> wrote:
>>
>> > Bump up this thread after 6 months.
>> >
>> > Is anyone still interested in the 2.9 release line? Or are we good to
>> start
>> > the EOL process? The 2.9.2 was released in Nov 2018.
>> >
>> > I'd really like to see the community to converge to fewer release lines
>> and
>> > make more frequent releases in each line.
>> >
>> > Thanks,
>> > Weichiu
>> >
>> >
>> > On Fri, Mar 6, 2020 at 5:47 PM Wei-Chiu Chuang 
>> wrote:
>> >
>> > > I think that's a great suggestion.
>> > > Currently, we make 1 minor release per year, and within each minor
>> > release
>> > > we bring up 1 thousand to 2 thousand commits in it compared with the
>> > > previous one.
>> > > I can totally understand it is a big bite for users to swallow.
>> Having a
>> > > more frequent release cycle, plus LTS and non-LTS releases should help
>> > with
>> > > this. (Of course we will need to make the release preparation much
>> > easier,
>> > > which is currently a pain)
>> > >
>> > > I am happy to discuss the release model further in the dev ML. LTS
>> v.s.
>> > > non-LTS is one suggestion.
>> > >
>> > > Another similar issue: In the past Hadoop strived to
>> > > maintain compatibility. However, this is no longer sustainable as more
>> > CVEs
>> > > coming from our dependencies: netty, jetty, jackson ... etc.
>> > > In many cases, updating the dependencies brings breaking changes. More
>> > > recently, especially in Hadoop 3.x, I started to make the effort to
>> > update
>> > > dependencies much more frequently. How do users feel about this
>> change?
>> > >
>> > > On Thu, Mar 5, 2020 at 7:58 AM Igor Dvorzhak 
>> > > wrote:
>> > >
>> > >> Maybe Hadoop will benefit from adopting a similar release and support
>> > >> strategy as Java? I.e. designate some releases as LTS and support
>> them
>> > for
>> > >> 2 (?) years (it seems that 2.7.x branch was de-facto LTS), other
>> non-LTS
>> > >> releases will be supported for 6 months (or until next release). This
>> > >> should allow to reduce maintenance cost of non-LTS release and
>> provide
>> > >> conservative users desired stability by allowing them to wait for new
>> > LTS
>> > >> release and upgrading to it.
>> > >>
>> > >> On Thu, Mar 5, 2020 at 1:26 AM Rupert Mazzucco <
>> > rupert.mazzu...@gmail.com>
>> > >> wrote:
>> > >>
>> > >>> After recently jumping from 2.7.7 to 2.10 without issue myself, I
>> vote
>> > >>> for keeping only the 2.10 line.
>> > >>> It would seem all other 2.x branches can upgrade to a 2.10.x easily
>> if
>> > >>> they feel like upgrading at all,
>> > >>> unlike a jump to 3.x, which may require more planning.
>> > >>>
>> > >>> I also vote for having only one main 3.x branch. Why are there 3.1.x
>> > and
>> > >>> 3.2.x seemingly competing,
>> > >>> and now 3.3.x? For a community that does not have the resources to
>> > >>> manage multiple release lines,
>> > >>> you guys sure like to multiply release lines a lot.
>> > >>>
>> > >>> Cheers
>> > >>> Rupert
>> > >>>
>> > >>> Am Mi., 4. März 2020 um 19:40 Uhr schrieb Wei-Chiu Chuang
>> > >>> :
>> > >>>
>> >  Forwarding the discussion thread from the dev mailing lists to the
>> > user
>> >  mailing lists.
>> > 
>> >  I'd like to get an idea of how many users are still on Hadoop 2.9.
>> >  Please share your thoughts.
>> > 
>> >  On Mon, Mar 2, 2020 at 6:30 PM Sree Vaddi
>> >   wrote:
>> > 
>> > > +1
>> > >
>> > > Sent from Yahoo Mail on Android
>> > >
>> > >   On Mon, Mar 2, 2020 at 5:12 PM, Wei-Chiu Chuang<
>> weic...@apache.org
>> > >
>> > > wrote:   Hi,
>> > >
>> > > Following the discussion to end branch-2.8, I want to start a
>> > > discussion
>> > > around what's next with branch-2.9. I am hesitant to use the word
>> > "end
>> > > of
>> > > life" but consider these facts:
>> > >
>> > > * 2.9.0 was released Dec 17, 2017.
>> > > * 2.9.2, the last 2.9.x release, went out Nov 19 2018, which is
>> more
>> > > than
>> > > 15 months ago.
>> > > * no one seems to be interested in being the release manager for
>> > 2.9.3.
>> > > * Most if not all of the active Hadoop contributors are using
>> Hadoop
>> > > 2.10
>> > > or Hadoop 3.x.
>> > > * We as a community do not have the cycle to manage multiple
>> release
>> > > line,
>> > > especially since Hadoop 3.3.0 is coming out soon.
>> > >
>> > > It is perhaps the time to gradually reduce our footprint in Hadoop
>> > > 2.x, and
>> > > encourage people to upgrade to Hadoop 3.x
>> > >
>> > 

Re: [DISCUSS] Hadoop 3.2.2 release

[Sunil Govindan]

Hi Xiaoqiao,

Thanks for volunteering.
I could help to guide you through the process.

Thanks,
Sunil

On Thu, Sep 3, 2020 at 9:09 AM Xiaoqiao He  wrote:

> Thanks Wei-Chiu for initiating this.
>
> It's been nearly a year since Hadoop-3.2.1 released(2019 Sep 22).
>
> We have several important fixes landed in branch-3.2 (around 37
> blockers/critical issues)[1].
> There are also 3 blockers/critical pending issues and 3 other pending
> issues on 3.2.2[2].
>
> I am willing to volunteer with the Hadoop 3.3.2 release work. Anyone who
> would like to contribute or offer some suggestions & release guides will
> be appreciated.
>
> Please let me know if you have any thoughts or comments on this plan.
>
> [1]
>
> https://issues.apache.org/jira/browse/YARN-10347?jql=project%20in%20(YARN%2C%20HDFS%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20status%20%3D%20Resolved%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20fixVersion%20%3D%203.2.2
> [2]
>
> https://issues.apache.org/jira/browse/YARN-10345?jql=project%20in%20(YARN%2C%20HDFS%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22Patch%20Available%22)%20AND%20affectedVersion%20%3D%203.2.2
>
> Thanks,
> He Xiaoqiao
>
>
> On Wed, Sep 2, 2020 at 7:27 AM Wei-Chiu Chuang  wrote:
>
> > Hi folks,
> >
> > I was reminded by Xiaoqiao that Hadoop 3.2.1 was made almost a year ago
> > (released on September 22, 2019) and we're overdue for a follow-up.
> >
> > @Rohith Sharma K S   you were the RM for
> > Hadoop 3.2.1. Are we planning to make the 3.2.2 releases soon? Xiaoqiao
> > wants to help with the release.
> >
> > Thanks,
> > Weichiu
> >
>

Re: Two more binding votes required for Apache Hadoop 2.10.1 RC0

[Sunil Govindan]

Thanks.
I will verify the release.

Sunil

On Sat, Sep 19, 2020 at 12:26 AM Wei-Chiu Chuang  wrote:

> Masatake is doing a great job rolling out 2.10.1 RC0. Let's give him a
> final push to get it out.
>
> Thanks,
> Wei-Chiu
>

Re: [VOTE] Release Apache Hadoop 2.10.1 (RC0)

[Sunil Govindan]

+1 (binding)

- verified checksum and sign. Shows as a Good signature from "Masatake
Iwasaki (CODE SIGNING KEY) "
- built from source
- ran basic MR job and looks good
- UI also seems fine

Thanks,
Sunil

On Sun, Sep 20, 2020 at 11:38 AM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> The RC0 got 2 binding +1's and 2 non-binging +1's [1].
>
> Based on the discussion about release vote [2],
> bylaws[3] defines the periods in minimum terms.
> We can extend it if there is not enough activity.
>
> I would like to extend the period to 7 days,
> until Monday September 21 at 10:00 am PDT.
>
> I will appreciate additional votes.
>
> Thanks,
> Masatake Iwasaki
>
> [1]
> https://lists.apache.org/thread.html/r16a7f36315a0673c7d522c41065e7ef9c9ee15c76ffcb5db80931002%40%3Ccommon-dev.hadoop.apache.org%3E
> [2]
> https://lists.apache.org/thread.html/e392b902273ee0c14ba34d72c44630e05f54cb3976109af510592ea2%401403330080%40%3Ccommon-dev.hadoop.apache.org%3E
> [3] https://hadoop.apache.org/bylaws.html
>
> On 2020/09/15 2:59, Masatake Iwasaki wrote:
> > Hi folks,
> >
> > This is the first release candidate for the second release of Apache
> Hadoop 2.10.
> > It contains 218 fixes/improvements since 2.10.0 [1].
> >
> > The RC0 artifacts are at:
> > http://home.apache.org/~iwasakims/hadoop-2.10.1-RC0/
> >
> > RC tag is release-2.10.1-RC0:
> > https://github.com/apache/hadoop/tree/release-2.10.1-RC0
> >
> > The maven artifacts are hosted here:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1279/
> >
> > My public key is available here:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > The vote will run for 5 days, until Saturday, September 19 at 10:00 am
> PDT.
> >
> > [1]
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.1
> >
> > Thanks,
> > Masatake Iwasaki
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

Re: Hadoop 3.2.2 Release Code Freeze Plan

[Sunil Govindan]

Thank you

Did we also cut the branch already ?

Sunil

On Fri, 16 Oct 2020 at 9:14 AM, Xiaoqiao He  wrote:

> Hi All,
>
> All issues targeted to 3.2.2 have been resolved from now on[1], Thanks
> everyone for your works.
> Will try to operate code frozen and create the release candidate today.
> Thanks.
>
> [1]
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>
> On Fri, Oct 2, 2020 at 2:34 PM Xiaoqiao He  wrote:
>
>> Thanks to Wei-Chiu for your reminder, the dashboard and filters have been
>> updated. It is visible to the public now.
>>
>> Thanks, Regards.
>>
>> [1]
>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>>
>>
>> On Fri, Oct 2, 2020 at 6:30 AM Wei-Chiu Chuang 
>> wrote:
>>
>>> Thanks Xiaoqiao!
>>> Glad to see this is moving along. I noticed your dashboard has private
>>> filters and therefore the results are not visible publicly.
>>>
>>> On Tue, Sep 29, 2020 at 10:02 PM Xiaoqiao He 
>>> wrote:
>>>
>>> > Hi All,
>>> >
>>> > Plan to code frozen for Hadoop-3.2.2 release at 2020/10/15. From now
>>> on,
>>> > most of the issues have been resolved. And there are two JIRA still
>>> open
>>> > which target to 3.2.2 traced by [1].
>>> >
>>> > * https://issues.apache.org/jira/browse/YARN-10244
>>> > * https://issues.apache.org/jira/browse/HADOOP-17287
>>> >
>>> > Please let us know if this is really blocking for 3.2.2, if not kindly
>>> > move it out. If required to involve in 3.2.2, Please try to push them
>>> > forward recently.
>>> >
>>> > Thanks & Best Regards,
>>> > He Xiaoqiao
>>> >
>>> > [1]
>>> >
>>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>>> >
>>>
>>

Re: Hadoop 3.2.2 Release Code Freeze Plan

[Sunil Govindan]

Looks good to me.

Thanks Xiaoquio.

+ Sunil

On Fri, Oct 16, 2020 at 12:36 PM Xiaoqiao He  wrote:

> Thanks Sunil for your reminder,
>
> branch-3.2.2 is ready now [1] and also mark branch-3.2 to 'prepare for
> 3.2.3 development' [2].
> Please help to give another check if you have time.
>
> Thanks.
>
> [1] https://github.com/apache/hadoop/commits/branch-3.2.2
> [2] https://github.com/apache/hadoop/commits/branch-3.2
>
> On Fri, Oct 16, 2020 at 12:03 PM Sunil Govindan  wrote:
>
>> Thank you
>>
>> Did we also cut the branch already ?
>>
>> Sunil
>>
>> On Fri, 16 Oct 2020 at 9:14 AM, Xiaoqiao He 
>> wrote:
>>
>>> Hi All,
>>>
>>> All issues targeted to 3.2.2 have been resolved from now on[1], Thanks
>>> everyone for your works.
>>> Will try to operate code frozen and create the release candidate today.
>>> Thanks.
>>>
>>> [1]
>>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>>>
>>> On Fri, Oct 2, 2020 at 2:34 PM Xiaoqiao He 
>>> wrote:
>>>
>>>> Thanks to Wei-Chiu for your reminder, the dashboard and filters have
>>>> been updated. It is visible to the public now.
>>>>
>>>> Thanks, Regards.
>>>>
>>>> [1]
>>>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>>>>
>>>>
>>>> On Fri, Oct 2, 2020 at 6:30 AM Wei-Chiu Chuang 
>>>> wrote:
>>>>
>>>>> Thanks Xiaoqiao!
>>>>> Glad to see this is moving along. I noticed your dashboard has private
>>>>> filters and therefore the results are not visible publicly.
>>>>>
>>>>> On Tue, Sep 29, 2020 at 10:02 PM Xiaoqiao He 
>>>>> wrote:
>>>>>
>>>>> > Hi All,
>>>>> >
>>>>> > Plan to code frozen for Hadoop-3.2.2 release at 2020/10/15. From now
>>>>> on,
>>>>> > most of the issues have been resolved. And there are two JIRA still
>>>>> open
>>>>> > which target to 3.2.2 traced by [1].
>>>>> >
>>>>> > * https://issues.apache.org/jira/browse/YARN-10244
>>>>> > * https://issues.apache.org/jira/browse/HADOOP-17287
>>>>> >
>>>>> > Please let us know if this is really blocking for 3.2.2, if not
>>>>> kindly
>>>>> > move it out. If required to involve in 3.2.2, Please try to push them
>>>>> > forward recently.
>>>>> >
>>>>> > Thanks & Best Regards,
>>>>> > He Xiaoqiao
>>>>> >
>>>>> > [1]
>>>>> >
>>>>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>>>>> >
>>>>>
>>>>

Re: [E] Re: Hadoop 3.2.2 Release Code Freeze Plan

[Sunil Govindan]

Yes. I ran a compilation and it also failed. This is an issue when we do
maven set version command because hadoop.version in the parent pom also
needs to be set.

3.2.2-SNAPSHOT

On Sat, Oct 17, 2020 at 12:15 AM Eric Badger 
wrote:

> Hi Xiaoqiao,
>
> I believe that
> https://github.com/apache/hadoop/commit/3274fd139d9b612e449fc234f8804a2a97ae6c47
> broke compilation for branch-3.2. Looks like you missed a 3.2.2 -> 3.2.3 in
> pom.xml in the top-level directory.
>
>
> https://github.com/apache/hadoop/blob/3274fd139d9b612e449fc234f8804a2a97ae6c47/pom.xml#L83
>
> Eric
>
> On Fri, Oct 16, 2020 at 6:43 AM Sunil Govindan  wrote:
>
>> Looks good to me.
>>
>> Thanks Xiaoquio.
>>
>> + Sunil
>>
>> On Fri, Oct 16, 2020 at 12:36 PM Xiaoqiao He 
>> wrote:
>>
>> > Thanks Sunil for your reminder,
>> >
>> > branch-3.2.2 is ready now [1] and also mark branch-3.2 to 'prepare for
>> > 3.2.3 development' [2].
>> > Please help to give another check if you have time.
>> >
>> > Thanks.
>> >
>> > [1]
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_hadoop_commits_branch-2D3.2.2&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=KVdP1SUmHYb-tZP8tcigmw&m=QopE4z-adN2ofsasgUJBO9T6otSqQwA_lvjlDvzaDj4&s=5VhJo-jLXCdf8Db8WO06oowMfEFT7pIio255mBm1Flg&e=
>> > [2]
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_hadoop_commits_branch-2D3.2&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=KVdP1SUmHYb-tZP8tcigmw&m=QopE4z-adN2ofsasgUJBO9T6otSqQwA_lvjlDvzaDj4&s=zESRGqLVBSaekRIRFHYT2n5LLpVQNoVEZGIz5WRLRFw&e=
>> >
>> > On Fri, Oct 16, 2020 at 12:03 PM Sunil Govindan 
>> wrote:
>> >
>> >> Thank you
>> >>
>> >> Did we also cut the branch already ?
>> >>
>> >> Sunil
>> >>
>> >> On Fri, 16 Oct 2020 at 9:14 AM, Xiaoqiao He 
>> >> wrote:
>> >>
>> >>> Hi All,
>> >>>
>> >>> All issues targeted to 3.2.2 have been resolved from now on[1], Thanks
>> >>> everyone for your works.
>> >>> Will try to operate code frozen and create the release candidate
>> today.
>> >>> Thanks.
>> >>>
>> >>> [1]
>> >>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_Dashboard.jspa-3FselectPageId-3D12335948&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=KVdP1SUmHYb-tZP8tcigmw&m=QopE4z-adN2ofsasgUJBO9T6otSqQwA_lvjlDvzaDj4&s=eACU__Fq-l9QTTfNhg2QTdUGZd5EnoEF_GoZVX7SVus&e=
>> >>>
>> >>> On Fri, Oct 2, 2020 at 2:34 PM Xiaoqiao He 
>> >>> wrote:
>> >>>
>> >>>> Thanks to Wei-Chiu for your reminder, the dashboard and filters have
>> >>>> been updated. It is visible to the public now.
>> >>>>
>> >>>> Thanks, Regards.
>> >>>>
>> >>>> [1]
>> >>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_Dashboard.jspa-3FselectPageId-3D12335948&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=KVdP1SUmHYb-tZP8tcigmw&m=QopE4z-adN2ofsasgUJBO9T6otSqQwA_lvjlDvzaDj4&s=eACU__Fq-l9QTTfNhg2QTdUGZd5EnoEF_GoZVX7SVus&e=
>> >>>>
>> >>>>
>> >>>> On Fri, Oct 2, 2020 at 6:30 AM Wei-Chiu Chuang 
>> >>>> wrote:
>> >>>>
>> >>>>> Thanks Xiaoqiao!
>> >>>>> Glad to see this is moving along. I noticed your dashboard has
>> private
>> >>>>> filters and therefore the results are not visible publicly.
>> >>>>>
>> >>>>> On Tue, Sep 29, 2020 at 10:02 PM Xiaoqiao He > >
>> >>>>> wrote:
>> >>>>>
>> >>>>> > Hi All,
>> >>>>> >
>> >>>>> > Plan to code frozen for Hadoop-3.2.2 release at 2020/10/15. From
>> now
>> >>>>> on,
>> >>>>> > most of the issues have been resolved. And there are two JIRA
>> still
>> >>>>> open
>> >>>>> > which target to 3.2.2 traced by [1].
>> >>>>> >
>> >>>>> > *
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_YARN-2D10244&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=KVdP1SUmHYb-tZP8tcigmw&m=QopE4z-adN2ofsasgUJBO9T6otSqQwA_lvjlDvzaDj4&s=DMkkzyzTBFbE3eek-Rn9bNABvCEVAp4d3KIhrvuWEc8&e=
>> >>>>> > *
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_HADOOP-2D17287&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=KVdP1SUmHYb-tZP8tcigmw&m=QopE4z-adN2ofsasgUJBO9T6otSqQwA_lvjlDvzaDj4&s=BadkXOH-nNJiIeBOZSmp9LyKiO699BcqPqkzr-_MT14&e=
>> >>>>> >
>> >>>>> > Please let us know if this is really blocking for 3.2.2, if not
>> >>>>> kindly
>> >>>>> > move it out. If required to involve in 3.2.2, Please try to push
>> them
>> >>>>> > forward recently.
>> >>>>> >
>> >>>>> > Thanks & Best Regards,
>> >>>>> > He Xiaoqiao
>> >>>>> >
>> >>>>> > [1]
>> >>>>> >
>> >>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_Dashboard.jspa-3FselectPageId-3D12335948&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=KVdP1SUmHYb-tZP8tcigmw&m=QopE4z-adN2ofsasgUJBO9T6otSqQwA_lvjlDvzaDj4&s=eACU__Fq-l9QTTfNhg2QTdUGZd5EnoEF_GoZVX7SVus&e=
>> >>>>> >
>> >>>>>
>> >>>>
>>
>

Re: [E] Re: Hadoop 3.2.2 Release Code Freeze Plan

[Sunil Govindan]

HI Xiaoqiao
Sorry, i missed ur messages earlier as it was night time.

But meet problems when executing step 11 of `Creating the release candidate
(X.Y.Z-RC)`. I am not sure what resources I should upload to sftp and
how to generate them. Any instructions wiki to follow up? thanks.
> *sftp home.apache.org <http://home.apache.org>*, so u need to have an
account in the home.apache.org. Usually only committers are having that
space. I ll double check and let you know



On Sat, Oct 17, 2020 at 10:14 PM Xiaoqiao He  wrote:

> Thanks Steven for your response.
>
> When I traced YARN-8990 and YARN-8992 I found that,
> a. for YARN-8992, it's fix version is set incorrectly, it seems not in
> branch-3.2 and all point branches but mark the fix version to branch-3.2.0.
> b. for YARN-8990, it is only in branch-3.2.0 but not in branch-3.2
> and correlative point branches.
> I am not familiar with YARN sub-project, but if we need to involve them
> into branch-3.2.2 and it is standard operation, I would like to wait until
> they are merged and prepare another release candidate.
> Thanks Steve again.
>
> Best,
> - He Xiaoqiao
>
> On Sat, Oct 17, 2020 at 10:45 PM Steven Rand 
> wrote:
>
> > Sorry for raising this so late, but I noticed just now that we never
> > managed to backport YARN-8990 to branch-3.2, so it's not in branch-3.2.2
> > either.
> >
> > That fix wasn't included in 3.2.1, so it's not a regression if we also
> > don't include it in 3.2.2. It's a blocking issue though, so I think it'd
> be
> > unfortunate if we missed this chance to include it in a 3.2.x maintenance
> > release. (Same for YARN-8992, which is a related JIRA.)
> >
> > Best,
> > Steve
> >
> > On Sat, Oct 17, 2020 at 12:25 PM Xiaoqiao He 
> > wrote:
> >
> >> Thanks Eric and Sunil for your reviews.
> >>
> >> Update pom.xml in the top-level directory.[1]
> >>
> >> > -
> >> 3.2.2-SNAPSHOT
> >> > +
> >>  3.2.3-SNAPSHOT
> >>
> >>
> >> Try to grep '3.2.2-SNAPSHOT' in the whole project and not found anymore,
> >> try to compile branch-3.2 at local and pass.
> >> Please give another review if convenient.
> >>
> >> Thanks. Regards.
> >>
> >> - He Xiaoqiao
> >>
> >> [1]
> >>
> >>
> https://github.com/apache/hadoop/commit/4a1f1bf5de060f4847c070c4fe920cc233feb721
> >>
> >> On Sat, Oct 17, 2020 at 10:49 AM Sunil Govindan 
> >> wrote:
> >>
> >> > Yes. I ran a compilation and it also failed. This is an issue when we
> do
> >> > maven set version command because hadoop.version in the parent pom
> also
> >> > needs to be set.
> >> >
> >> > 3.2.2-SNAPSHOT
> >> >
> >> > On Sat, Oct 17, 2020 at 12:15 AM Eric Badger <
> ebad...@verizonmedia.com>
> >> > wrote:
> >> >
> >> >> Hi Xiaoqiao,
> >> >>
> >> >> I believe that
> >> >>
> >>
> https://github.com/apache/hadoop/commit/3274fd139d9b612e449fc234f8804a2a97ae6c47
> >> >> broke compilation for branch-3.2. Looks like you missed a 3.2.2 ->
> >> 3.2.3 in
> >> >> pom.xml in the top-level directory.
> >> >>
> >> >>
> >> >>
> >>
> https://github.com/apache/hadoop/blob/3274fd139d9b612e449fc234f8804a2a97ae6c47/pom.xml#L83
> >> >>
> >> >> Eric
> >> >>
> >> >> On Fri, Oct 16, 2020 at 6:43 AM Sunil Govindan 
> >> wrote:
> >> >>
> >> >>> Looks good to me.
> >> >>>
> >> >>> Thanks Xiaoquio.
> >> >>>
> >> >>> + Sunil
> >> >>>
> >> >>> On Fri, Oct 16, 2020 at 12:36 PM Xiaoqiao He  >
> >> >>> wrote:
> >> >>>
> >> >>> > Thanks Sunil for your reminder,
> >> >>> >
> >> >>> > branch-3.2.2 is ready now [1] and also mark branch-3.2 to 'prepare
> >> for
> >> >>> > 3.2.3 development' [2].
> >> >>> > Please help to give another check if you have time.
> >> >>> >
> >> >>> > Thanks.
> >> >>> >
> >> >>> > [1]
> >> >>>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_hadoop_commits_branch-2D3.2.2&d=DwIBaQ&

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC1

[Sunil Govindan]

Thanks Akira. I just saw that the revert of HADOOP-17306 is now pushed to
the 3.2.2 branch as well.

Let's do one more RC. Thanks @Xiaoqiao He  for
helping.

Thanks
Sunil

On Fri, Nov 6, 2020 at 6:58 AM Akira Ajisaka  wrote:

> -1
>
> - YARN resource localization is broken by HADOOP-17306 and it has been
> reverted. It should be reverted from 3.2.2 as well. (Thank you Jim
> Brennan for the report!)
> - Would you include HDFS-15643 in RC2? This fixes checksum error in EC
> with ISA-L.
>
> Thank you He Xiaoqiao for preparing the release candidates.
>
> -Akira
>
> On Tue, Nov 3, 2020 at 7:48 PM Xiaoqiao He  wrote:
> >
> > Hi folks,
> >
> > I have put together a release candidate (RC1) for Hadoop-3.2.2.
> >
> > It contains *473[1]* fixed jira issues since 3.2.1 release which also
> > include many features and improvements(Please reference the full set of
> > release notes).
> >
> > The RC is available at:
> > http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC1
> > The RC tag in github is here:
> > https://github.com/apache/hadoop/tree/release-3.2.2-RC1
> > The maven artifacts are staged at:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1286
> >
> > You can find my public key at:
> > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS or
> > https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
> >
> > Please try the release and vote. The vote will run for 5 days until
> > 2020/11/09 at 00:00 CST.
> >
> > I have done a simple testing with my pseudo cluster.
> > * Verify gpg sign and md5sum.
> > * Setup pseudo cluster with HDFS and YARN.
> > * Run simple Shell/Job.
> > * Check the web UI of
> > NameNode/DFSRouter/DataNode/Resourcemanager/NodeManager.
> > My +1 to start.
> >
> > NOTE:
> > A. Explanation about RC1 rather than RC0. After code freeze for RC0 there
> > are several new critical issues merged so create RC1.
> > B. Thanks Sunil/Wei-Chiu/Sammi Chen/Eric Badger for your very helpful
> guide.
> >
> > Thanks
> > He Xiaoqiao
> >
> > [1]
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4

[Sunil Govindan]

Hi Xiaoqiao and Wei-chiu

I am a bit confused after seeing both *.sha512 and *.md5 files in the RC
directory.
Are we releasing both now?

Thanks
Sunil

On Wed, Dec 9, 2020 at 10:32 PM Xiaoqiao He  wrote:

> Hi folks,
>
> The release candidate (RC4) for Hadoop-3.2.2 is available now.
> There are 10 commits[1] differences between RC4 and RC3[2].
>
> The RC4 is available at:
> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC4
> The RC4 tag in github is here:
> https://github.com/apache/hadoop/tree/release-3.2.2-RC4
> The maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1296
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
>
> Please try the release and vote.
>
> I have done a simple testing with my pseudo cluster.
> * Verify gpg sign and md5sum.
> * Setup pseudo cluster with HDFS and YARN.
> * Run simple FsShell - mkdir/put/get/rename/mv.
> * Submit example mr job and check the result - Pi/wordcount.
> * Check the web UI of NameNode/DataNode/Resourcemanager/NodeManager.
> My +1 to start.
>
> Thanks,
> He Xiaoqiao
>
> [1]
>
> https://github.com/apache/hadoop/compare/release-3.2.2-RC3...release-3.2.2-RC4
> [2]
>
> https://lists.apache.org/thread.html/rfb74c3a5d4f223c5804d8ee622829263740cd8701c8f3fc8b6f970af%40%3Chdfs-dev.hadoop.apache.org%3E
> [3]
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4

[Sunil Govindan]

Reference:
https://www.apache.org/dev/release-distribution.html#sigs-and-sums
Also, we had a Jira to track this HADOOP-15930
<https://issues.apache.org/jira/browse/HADOOP-15930>.

Thanks
Sunil

On Sat, Dec 19, 2020 at 10:16 AM Sunil Govindan  wrote:

> Hi Xiaoqiao and Wei-chiu
>
> I am a bit confused after seeing both *.sha512 and *.md5 files in the RC
> directory.
> Are we releasing both now?
>
> Thanks
> Sunil
>
> On Wed, Dec 9, 2020 at 10:32 PM Xiaoqiao He  wrote:
>
>> Hi folks,
>>
>> The release candidate (RC4) for Hadoop-3.2.2 is available now.
>> There are 10 commits[1] differences between RC4 and RC3[2].
>>
>> The RC4 is available at:
>> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC4
>> The RC4 tag in github is here:
>> https://github.com/apache/hadoop/tree/release-3.2.2-RC4
>> The maven artifacts are staged at:
>> https://repository.apache.org/content/repositories/orgapachehadoop-1296
>>
>> You can find my public key at:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
>> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
>>
>> Please try the release and vote.
>>
>> I have done a simple testing with my pseudo cluster.
>> * Verify gpg sign and md5sum.
>> * Setup pseudo cluster with HDFS and YARN.
>> * Run simple FsShell - mkdir/put/get/rename/mv.
>> * Submit example mr job and check the result - Pi/wordcount.
>> * Check the web UI of NameNode/DataNode/Resourcemanager/NodeManager.
>> My +1 to start.
>>
>> Thanks,
>> He Xiaoqiao
>>
>> [1]
>>
>> https://github.com/apache/hadoop/compare/release-3.2.2-RC3...release-3.2.2-RC4
>> [2]
>>
>> https://lists.apache.org/thread.html/rfb74c3a5d4f223c5804d8ee622829263740cd8701c8f3fc8b6f970af%40%3Chdfs-dev.hadoop.apache.org%3E
>> [3]
>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>>
>

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4

[Sunil Govindan]

Hi Xiaoqiao,

Please remove the md5 files from your shared RC4 repo. Thanks, @Akira
Ajisaka  for sharing this input.

Thanks
Sunil

On Sat, Dec 19, 2020 at 10:21 AM Sunil Govindan  wrote:

> Reference:
> https://www.apache.org/dev/release-distribution.html#sigs-and-sums
> Also, we had a Jira to track this HADOOP-15930
> <https://issues.apache.org/jira/browse/HADOOP-15930>.
>
> Thanks
> Sunil
>
> On Sat, Dec 19, 2020 at 10:16 AM Sunil Govindan  wrote:
>
>> Hi Xiaoqiao and Wei-chiu
>>
>> I am a bit confused after seeing both *.sha512 and *.md5 files in the RC
>> directory.
>> Are we releasing both now?
>>
>> Thanks
>> Sunil
>>
>> On Wed, Dec 9, 2020 at 10:32 PM Xiaoqiao He 
>> wrote:
>>
>>> Hi folks,
>>>
>>> The release candidate (RC4) for Hadoop-3.2.2 is available now.
>>> There are 10 commits[1] differences between RC4 and RC3[2].
>>>
>>> The RC4 is available at:
>>> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC4
>>> The RC4 tag in github is here:
>>> https://github.com/apache/hadoop/tree/release-3.2.2-RC4
>>> The maven artifacts are staged at:
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1296
>>>
>>> You can find my public key at:
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
>>> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
>>>
>>> Please try the release and vote.
>>>
>>> I have done a simple testing with my pseudo cluster.
>>> * Verify gpg sign and md5sum.
>>> * Setup pseudo cluster with HDFS and YARN.
>>> * Run simple FsShell - mkdir/put/get/rename/mv.
>>> * Submit example mr job and check the result - Pi/wordcount.
>>> * Check the web UI of NameNode/DataNode/Resourcemanager/NodeManager.
>>> My +1 to start.
>>>
>>> Thanks,
>>> He Xiaoqiao
>>>
>>> [1]
>>>
>>> https://github.com/apache/hadoop/compare/release-3.2.2-RC3...release-3.2.2-RC4
>>> [2]
>>>
>>> https://lists.apache.org/thread.html/rfb74c3a5d4f223c5804d8ee622829263740cd8701c8f3fc8b6f970af%40%3Chdfs-dev.hadoop.apache.org%3E
>>> [3]
>>>
>>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>>>
>>

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4

[Sunil Govindan]

Thanks, Xiaoqiao.
All files are looking good.

However, while I did the tests to verify the RC, I ran into a serious NPE
in YARN.
I raised YARN-10540 <https://issues.apache.org/jira/browse/YARN-10540> to
analyze this further. I think this issue due to YARN-10450
<https://issues.apache.org/jira/browse/YARN-10450>.
In the trunk, I am not able to see this issue. So It could be possible that
some patches are not backported to branch-3.2.2.

UI1 & UI2 nodes page is not working at this moment. I will check a bit more
to see about this and update here.

Thanks
Sunil

On Sat, Dec 19, 2020 at 5:36 PM Xiaoqiao He  wrote:

> Thanks Sunil, md5 files have been removed from RC4. Please have a look.
> Thanks & Regards.
>
> - He Xiaoqiao
>
> On Sat, Dec 19, 2020 at 7:22 PM Sunil Govindan  wrote:
>
>> Hi Xiaoqiao,
>>
>> Please remove the md5 files from your shared RC4 repo. Thanks, @Akira
>> Ajisaka  for sharing this input.
>>
>> Thanks
>> Sunil
>>
>> On Sat, Dec 19, 2020 at 10:21 AM Sunil Govindan 
>> wrote:
>>
>>> Reference:
>>> https://www.apache.org/dev/release-distribution.html#sigs-and-sums
>>> Also, we had a Jira to track this HADOOP-15930
>>> <https://issues.apache.org/jira/browse/HADOOP-15930>.
>>>
>>> Thanks
>>> Sunil
>>>
>>> On Sat, Dec 19, 2020 at 10:16 AM Sunil Govindan 
>>> wrote:
>>>
>>>> Hi Xiaoqiao and Wei-chiu
>>>>
>>>> I am a bit confused after seeing both *.sha512 and *.md5 files in the
>>>> RC directory.
>>>> Are we releasing both now?
>>>>
>>>> Thanks
>>>> Sunil
>>>>
>>>> On Wed, Dec 9, 2020 at 10:32 PM Xiaoqiao He 
>>>> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> The release candidate (RC4) for Hadoop-3.2.2 is available now.
>>>>> There are 10 commits[1] differences between RC4 and RC3[2].
>>>>>
>>>>> The RC4 is available at:
>>>>> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC4
>>>>> The RC4 tag in github is here:
>>>>> https://github.com/apache/hadoop/tree/release-3.2.2-RC4
>>>>> The maven artifacts are staged at:
>>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1296
>>>>>
>>>>> You can find my public key at:
>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
>>>>> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
>>>>>
>>>>> Please try the release and vote.
>>>>>
>>>>> I have done a simple testing with my pseudo cluster.
>>>>> * Verify gpg sign and md5sum.
>>>>> * Setup pseudo cluster with HDFS and YARN.
>>>>> * Run simple FsShell - mkdir/put/get/rename/mv.
>>>>> * Submit example mr job and check the result - Pi/wordcount.
>>>>> * Check the web UI of NameNode/DataNode/Resourcemanager/NodeManager.
>>>>> My +1 to start.
>>>>>
>>>>> Thanks,
>>>>> He Xiaoqiao
>>>>>
>>>>> [1]
>>>>>
>>>>> https://github.com/apache/hadoop/compare/release-3.2.2-RC3...release-3.2.2-RC4
>>>>> [2]
>>>>>
>>>>> https://lists.apache.org/thread.html/rfb74c3a5d4f223c5804d8ee622829263740cd8701c8f3fc8b6f970af%40%3Chdfs-dev.hadoop.apache.org%3E
>>>>> [3]
>>>>>
>>>>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>>>>>
>>>>

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4

[Sunil Govindan]

I had some offline talks with a few folks.
This issue is happening only in Mac, hence ideally it does not cause much
of a problem in the supported OS.

I will wait for feedback here to see whether we need another RC by fixing
this. And will continue the discussion in the jira.

Thanks
Sunil

On Sat, Dec 19, 2020 at 11:07 PM Sunil Govindan  wrote:

> Thanks, Xiaoqiao.
> All files are looking good.
>
> However, while I did the tests to verify the RC, I ran into a serious NPE
> in YARN.
> I raised YARN-10540 <https://issues.apache.org/jira/browse/YARN-10540> to
> analyze this further. I think this issue due to YARN-10450
> <https://issues.apache.org/jira/browse/YARN-10450>.
> In the trunk, I am not able to see this issue. So It could be possible
> that some patches are not backported to branch-3.2.2.
>
> UI1 & UI2 nodes page is not working at this moment. I will check a bit
> more to see about this and update here.
>
> Thanks
> Sunil
>
> On Sat, Dec 19, 2020 at 5:36 PM Xiaoqiao He  wrote:
>
>> Thanks Sunil, md5 files have been removed from RC4. Please have a look.
>> Thanks & Regards.
>>
>> - He Xiaoqiao
>>
>> On Sat, Dec 19, 2020 at 7:22 PM Sunil Govindan  wrote:
>>
>>> Hi Xiaoqiao,
>>>
>>> Please remove the md5 files from your shared RC4 repo. Thanks, @Akira
>>> Ajisaka  for sharing this input.
>>>
>>> Thanks
>>> Sunil
>>>
>>> On Sat, Dec 19, 2020 at 10:21 AM Sunil Govindan 
>>> wrote:
>>>
>>>> Reference:
>>>> https://www.apache.org/dev/release-distribution.html#sigs-and-sums
>>>> Also, we had a Jira to track this HADOOP-15930
>>>> <https://issues.apache.org/jira/browse/HADOOP-15930>.
>>>>
>>>> Thanks
>>>> Sunil
>>>>
>>>> On Sat, Dec 19, 2020 at 10:16 AM Sunil Govindan 
>>>> wrote:
>>>>
>>>>> Hi Xiaoqiao and Wei-chiu
>>>>>
>>>>> I am a bit confused after seeing both *.sha512 and *.md5 files in the
>>>>> RC directory.
>>>>> Are we releasing both now?
>>>>>
>>>>> Thanks
>>>>> Sunil
>>>>>
>>>>> On Wed, Dec 9, 2020 at 10:32 PM Xiaoqiao He 
>>>>> wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> The release candidate (RC4) for Hadoop-3.2.2 is available now.
>>>>>> There are 10 commits[1] differences between RC4 and RC3[2].
>>>>>>
>>>>>> The RC4 is available at:
>>>>>> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC4
>>>>>> The RC4 tag in github is here:
>>>>>> https://github.com/apache/hadoop/tree/release-3.2.2-RC4
>>>>>> The maven artifacts are staged at:
>>>>>>
>>>>>> https://repository.apache.org/content/repositories/orgapachehadoop-1296
>>>>>>
>>>>>> You can find my public key at:
>>>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
>>>>>> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
>>>>>>
>>>>>> Please try the release and vote.
>>>>>>
>>>>>> I have done a simple testing with my pseudo cluster.
>>>>>> * Verify gpg sign and md5sum.
>>>>>> * Setup pseudo cluster with HDFS and YARN.
>>>>>> * Run simple FsShell - mkdir/put/get/rename/mv.
>>>>>> * Submit example mr job and check the result - Pi/wordcount.
>>>>>> * Check the web UI of NameNode/DataNode/Resourcemanager/NodeManager.
>>>>>> My +1 to start.
>>>>>>
>>>>>> Thanks,
>>>>>> He Xiaoqiao
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>> https://github.com/apache/hadoop/compare/release-3.2.2-RC3...release-3.2.2-RC4
>>>>>> [2]
>>>>>>
>>>>>> https://lists.apache.org/thread.html/rfb74c3a5d4f223c5804d8ee622829263740cd8701c8f3fc8b6f970af%40%3Chdfs-dev.hadoop.apache.org%3E
>>>>>> [3]
>>>>>>
>>>>>> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>>>>>>
>>>>>

Re: [E] Re: [VOTE] Release Apache Hadoop 3.2.2 - RC4

[Sunil Govindan]

Hi Xiaoqiao
This is not an issue. You are missing the CORS config in YARN. I have
updated those configurations in the jira.
I feel that we can call a new RC as YARN-10540 is backported already.

Thanks
Sunil

On Wed, Dec 23, 2020 at 3:54 PM Xiaoqiao He  wrote:

> Thanks Jim, Eric and Sunil,
>
> I tried to backport YARN-10540 to branch-3.2.2 at local and check yarn
> ui2, there is still one link that does not work correctly, But I am not
> sure if it is expected. Detailed information is attached at YARN-10540.
> If someone could confirm or update, please let me know. I would like to
> backport then and prepare another RC.
> NOTE: Hadoop-3.2.2 will not release as we plan because some expected
> issues are in progress. It will be postponed to 2021/01, and I will try my
> best to push it forward. AND Thanks to all reviewers (too numerous to
> mention).
>
> - He Xiaoqiao
>
>
> On Tue, Dec 22, 2020 at 8:32 AM Wei-Chiu Chuang 
> wrote:
>
>> We should target
>> 3.4.0,
>> 3.3.1
>> 3.2.3 (or 3.2.2)
>>
>> I updated and set 3.3.0 released in YARN.
>>
>>
>> Time to clean up some of the old releases. I'll do that.
>> On Mon, Dec 21, 2020 at 4:04 PM Eric Badger
>>  wrote:
>>
>>> I've committed https://issues.apache.org/jira/browse/YARN-10540.
>>> Xiaoqiao,
>>> feel free to cherry-pick this into the 3.2.2 release branch if you think
>>> it
>>> is relevant.
>>>
>>> Also, can someone tell me which releases we should be targeting?
>>> Currently
>>> these versions are all Unreleased on JIRA:
>>> 3.4.1, 3.4.0
>>> 3.3.1, 3.3.0
>>> 3.2.3, 3.2.2
>>>
>>> As far as I know, neither trunk nor 3.3 have a release going on. So I
>>> don't
>>> know why there are 2 versions as unreleased there.
>>>
>>> Eric
>>>
>>> On Mon, Dec 21, 2020 at 3:20 PM Jim Brennan
>>>  wrote:
>>>
>>> > I put up a patch for
>>> >
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_YARN-2D10540&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=KVdP1SUmHYb-tZP8tcigmw&m=WEMl9mt9SZiq7P20DPgeZO69TFwf3d0eldQGRWlLyQg&s=FdkrkiWqHwuibT6qFqvApTkO5aTimLx7WeP74tka5XM&e=
>>> > .
>>> > Thanks for bringing it to my attention.
>>> > Jim
>>> >
>>> > On Mon, Dec 21, 2020 at 10:36 AM Sunil Govindan 
>>> wrote:
>>> >
>>> > > I had some offline talks with a few folks.
>>> > > This issue is happening only in Mac, hence ideally it does not cause
>>> much
>>> > > of a problem in the supported OS.
>>> > >
>>> > > I will wait for feedback here to see whether we need another RC by
>>> fixing
>>> > > this. And will continue the discussion in the jira.
>>> > >
>>> > > Thanks
>>> > > Sunil
>>> > >
>>> > > On Sat, Dec 19, 2020 at 11:07 PM Sunil Govindan 
>>> > wrote:
>>> > >
>>> > > > Thanks, Xiaoqiao.
>>> > > > All files are looking good.
>>> > > >
>>> > > > However, while I did the tests to verify the RC, I ran into a
>>> serious
>>> > NPE
>>> > > > in YARN.
>>> > > > I raised YARN-10540 <
>>> > >
>>> >
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_YARN-2D10540&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=7Imi06B91L3gbxmt5ChzH4cwlA2_f2tmXh3OXmV9MLw&m=nSlLXPsCxZGl0VV03dBWreCNrSH0SsNAZzmjRWO-2Zg&s=8i-pN_j9VKNxmOzU6gYGtWm_IVyeZkBcMwVI2eyzpRk&e=
>>> > > > to
>>> > > > analyze this further. I think this issue due to YARN-10450
>>> > > > <
>>> > >
>>> >
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_YARN-2D10450&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=7Imi06B91L3gbxmt5ChzH4cwlA2_f2tmXh3OXmV9MLw&m=nSlLXPsCxZGl0VV03dBWreCNrSH0SsNAZzmjRWO-2Zg&s=QEHMGtEbBz5Gn7mW4UsGlc-wNZ8ugZwiFQBy2pTx-Fw&e=
>>> > > >.
>>> > > > In the trunk, I am not able to see this issue. So It could be
>>> possible
>>> > > > that some patches are not backported to branch-3.2.2.
>>> > > >
>>> > > > UI1 & UI2 nodes page is not working at this moment. I will check a
>>> bit
&

Re: [VOTE] Release Apache Hadoop 3.2.2 - RC5

[Sunil Govindan]

+1 (binding)

- Verified the checksums and signatures
- Built from source
- Ran a few MR jobs
- Verified RM UI

Thanks
Sunil

On Sun, Jan 3, 2021 at 8:10 PM Xiaoqiao He  wrote:

> Hi folks,
>
> The release candidate (RC5) for Hadoop-3.2.2 is available now.
> There are 4 commits[1] differences between RC5 and RC4[2].
>
> The RC5 is available at:
> http://people.apache.org/~hexiaoqiao/hadoop-3.2.2-RC5
> The RC5 tag in github is here:
> https://github.com/apache/hadoop/tree/release-3.2.2-RC5
> The maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1298
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS or
> https://people.apache.org/keys/committer/hexiaoqiao.asc directly.
>
> Please try the release and vote.
>
> I have done a simple test.
> * Verify gpg sign and md5sum.
> * Check staging repositories point to the correct artifact(
> https://repository.apache.org/content/repositories/staging/).
> * Setup pseudo cluster with HDFS and YARN.
> * Run simple FsShell - mkdir/put/get/mv/rm.
> * Submit example mr applications and check the result - Pi & wordcount.
> * Check the web UI and the release year of
> NameNode/DataNode/Resourcemanager/NodeManager including YARN UI2.
>
> My +1 to start.
>
> Thanks and Happy New Year!
> - He Xiaoqiao
>
> [1]
>
> https://github.com/apache/hadoop/compare/release-3.2.2-RC4...release-3.2.2-RC5
> [2]
>
> https://lists.apache.org/thread.html/r8911d2934ebfe3869874842be94d1cfb00d99334bc93819d71466243%40%3Chdfs-dev.hadoop.apache.org%3E
> [3]
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12335948
>

Re: [ANNOUNCE] Apache Hadoop 3.2.2 release

[Sunil Govindan]

Congratulations Hadoop Community!

Thanks, Xiaoqiao for leading and helping to make this release happen.
Really appreciate your efforts.

Thanks,
Sunil.

On Fri, Jan 15, 2021 at 1:38 PM Xiaoqiao He  wrote:

> Hi All,
>
> It gives me great pleasure to announce that the Apache Hadoop community has
> voted to release Apache Hadoop 3.2.2.
>
> Apache Hadoop 3.2.2 includes 516 fixes since the previous Hadoop 3.2.1
> release.
> * For major changes included in Hadoop 3.2.2, please refer to Hadoop 3.2.2
> main page [1].
> * For more details about fixes in 3.2.2 release, please read the changelog
> [2] and release notes [3].
> The release news is posted on the Hadoop website too, you can go to the
> downloads section directly [4].
>
> Many thanks to everyone who contributed to the release, and everyone in the
> Apache Hadoop community! This release is a direct result of your great
> contributions.
>
> Many thanks to everyone who helped in this release process!
>
> Many thanks to Sunil Govindan, Wei-Chiu Chuang, Chao Sun, Steve Loughran
> and other folks who continued helps for this release process.
>
> [1] https://hadoop.apache.org/docs/r3.2.2/
> [2]
>
> https://hadoop.apache.org/docs/r3.2.2/hadoop-project-dist/hadoop-common/release/3.2.2/CHANGELOG.3.2.2.html
> [3]
>
> https://hadoop.apache.org/docs/r3.2.2/hadoop-project-dist/hadoop-common/release/3.2.2/RELEASENOTES.3.2.2.html
> [4] https://hadoop.apache.org/releases.html
>
> Best Regards,
> He Xiaoqiao
>

Re: [DISCUSS] Hadoop 3.3.1 release

[Sunil Govindan]

Hi Wei-Chiu,

What will be the next steps here for 3.3.1 planning?

Thanks
Sunil

On Mon, Feb 8, 2021 at 11:56 PM Stack  wrote:

> On Wed, Feb 3, 2021 at 6:41 AM Steve Loughran  >
> wrote:
>
> >
> > Regarding blockers &c: how about we have a little hackathon where we try
> > and get things in. This means a promise of review time from the people
> with
> > commit rights and other people who understand the code (Stack?)
> >
> >
>
> I'm up for helping get 3.3.1 out (reviewing, hackathon, testing).
> Thanks,
> S
>
>
>
>
> > -steve
> >
> > On Thu, 28 Jan 2021 at 06:48, Ayush Saxena  wrote:
> >
> > > +1
> > > Just to mention we would need to release hadoop-thirdparty too before.
> > > Presently we are using the snapshot version of it.
> > >
> > > -Ayush
> > >
> > > > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang 
> > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Hadoop 3.3.0 was released half a year ago, and as of now we've
> > > accumulated
> > > > more than 400 changes in the branch-3.3. A number of downstreamers
> are
> > > > eagerly waiting for 3.3.1 which addresses the guava version conflict
> > > issue.
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20
> > > >
> > > > We should start the release work for 3.3.1 before the diff becomes
> even
> > > > larger.
> > > >
> > > > I believe there are  currently only two real blockers for a 3.3.1
> > (using
> > > > this filter
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC
> > > > )
> > > >
> > > >
> > > >   1. HDFS-15566 
> > > >   2.
> > > >  1. HADOOP-17112 <
> > https://issues.apache.org/jira/browse/HADOOP-17112
> > > >
> > > > 2.
> > > >
> > > >
> > > >
> > > > Is there anyone who would volunteer to be the 3.3.1 RM?
> > > >
> > > > Also, the HowToRelease wiki does not describe the ARM build process.
> > > That's
> > > > going to be important for future releases.
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Seeing YETUS issue in jenkins runs

[Sunil Govindan]

Hi All,

We are getting the below error in recent runs.

-1.  yetus.   0m 8s.   Unprocessed flag(s): --findbugs-strict-precheck

https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/636/console

Are there any knows issues or some workarounds?
Please advise.

Thanks
Sunil

Re: [DISCUSS] cull unused modules from under hadoop-tools

[Sunil Govindan]

hadoop-sls is used for perf tests.

Thanks
Sunil

On Thu, Apr 7, 2022 at 11:57 AM Owen O'Malley 
wrote:

> We really need input from the users as well. In trunk, hadoop-ozone and
> hadoop-ant are already gone.
>
> Used:
> hadoop-aliyun
> hadoop-aws
> hadoop-distcp
> hadoop-azure-datalake
> hadoop-azure
> hadoop-dynamometer
> hadoop-federation-balance
>
> Questions:
> hadoop-archive-logs
> hadoop-archives
> hadoop-datajoin
> hadoop-extras
> hadoop-fs2img
> hadoop-gridmix
> hadoop-kafka
> hadoop-openstack
> hadoop-pipes
> hadoop-resourceestimator
> hadoop-rumen
> hadoop-sls
> hadoop-streaming
>
> On Thu, Apr 7, 2022 at 3:17 PM Ayush Saxena  wrote:
>
> > Attic will not take this or will be very painfull for us, they in general
> > take the entire project. Had a word with infra folks as well, we can
> > probably move them under a seperate repo under hadoop and then archive
> it.
> > Depandabot and such don’t scan archived repos. This should also solve our
> > pourpose.
> > Does that make sense?
> >
> > -Ayush
> >
> > > On 07-Apr-2022, at 7:03 PM, Steve Loughran  >
> > wrote:
> > >
> > > do that and we still have to worry about CVEs, dynabot complaints etc,
> > > build breaking changes.
> > >
> > > I am being more ruthless: can we move these into some attic repo.
> > >
> > >> On Fri, 1 Apr 2022 at 09:39, Vinayakumar B 
> > wrote:
> > >>
> > >> Can move unused ones to separate repo under hadoop, and introduce a
> > >> separate independent release cycle in case required.
> > >>
> > >> Any thoughts on this?
> > >> -Vinay
> > >>
> > >>> On Fri, 1 Apr 2022 at 1:31 PM, Ayush Saxena 
> > wrote:
> > >>>
> > >>> Distcp (I have a use case for it)
> > >>> Hadoop-federation-balance(RBF uses it, we will take care in case it
> > >>> bothers)
> > >>> Hadoop-Dynamometer(This I feel is being used from jira activies)
> > >>>
> > >>> So, these three we should let stay as is.
> > >>> Others the object store ones are active. You know which all are
> needed.
> > >>> For rest we can call for a vote and drop them if everyone agrees.
> > >>>
> > >>> -Ayush
> > >>>
> >  On 31-Mar-2022, at 4:40 PM, Steve Loughran
> >  > >>>
> > >>> wrote:
> > 
> >   how many of the modules under hadoop-tools get used/maintained?
> > 
> >  hadoop-aliyun
> >  hadoop-ant
> >  hadoop-archive-logs
> >  hadoop-archives
> >  hadoop-aws
> >  hadoop-azure
> >  hadoop-azure-datalake
> >  hadoop-datajoin
> >  hadoop-distcp
> >  hadoop-dynamometer
> >  hadoop-extras
> >  hadoop-federation-balance
> >  hadoop-fs2img
> >  hadoop-ftp
> >  hadoop-gridmix
> >  hadoop-kafka
> >  hadoop-openstack
> >  hadoop-ozone
> >  hadoop-pipes
> >  hadoop-resourceestimator
> >  hadoop-rumen
> >  hadoop-sls
> >  hadoop-streaming
> > 
> >  I know distcp is universal, and the aws. azure, aliyun modules are
> >  active. hadoop-azure-datalake doesn't get maintenance, but it should
> > >> stay
> >  around until microsoft remove the gen1 ADLS service
> > 
> >  But what about all the others? the hadoop-openstack one hasn't been
> > >>> touched
> >  or tested for a few years, and IMO could be cut immediately. what
> > about
> >  others? does hadoop-streaming or hadoop-pipes get use any more?
> > 
> >  Existing code may use these, but having them in the codebase only
> > >> creates
> >  maintenance work, especially if security fixes need to go in on the
> > >> code
> > >>> or
> >  are caused by dependencies.
> > >>>
> > >>> -
> > >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >>>
> > >>> --
> > >> -Vinay
> > >>
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>

Re: [ANNOUNCE] New Hadoop PMC member - Mukund Thakur

[Sunil Govindan]

Congratulations Mukund!!!

Thanks
Sunil

On Wed, Feb 8, 2023 at 1:48 AM Vinayakumar B 
wrote:

> On behalf of the Apache Hadoop PMC, I am pleased to announce that Mukund
> Thakur (mthakur) has accepted the PMC's invitation to become a PMC member
> on
> the project. We appreciate all of Mukund’s generous contributions thus far
> and look forward to his continued involvement.
>
> Congratulations and welcome, Mukund !
>
> -Vinayakumar B
> --
> -Vinay
>

[jira] [Reopened] (HADOOP-15483) Upgrade jquery to version 3.3.1

[Sunil Govindan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil Govindan reopened HADOOP-15483:
-

Reopening to run jenkins

> Upgrade jquery to version 3.3.1
> ---
>
> Key: HADOOP-15483
> URL: https://issues.apache.org/jira/browse/HADOOP-15483
> Project: Hadoop Common
>  Issue Type: Task
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-15483-branch-3.1.001.patch, 
> HADOOP-15483.001.patch, HADOOP-15483.002.patch, HADOOP-15483.003.patch, 
> HADOOP-15483.004.patch, HADOOP-15483.005.patch, HADOOP-15483.006.patch, 
> HADOOP-15483.007.patch, HADOOP-15483.008.patch
>
>
> This Jira aims to upgrade jquery to version 3.3.1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15668) mvn test goal fails on HADOOP-15407 branch

[Sunil Govindan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil Govindan resolved HADOOP-15668.
-
Resolution: Not A Problem

Closed with correct status

> mvn test goal fails on HADOOP-15407 branch
> --
>
> Key: HADOOP-15668
> URL: https://issues.apache.org/jira/browse/HADOOP-15668
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>
> It's very easy to reproduce:
> {code}cd hadoop-common-project/hadoop-common/
> mvn test -Dtest=Whatever{code}
> The error is due to
> {code} [exec] Running bats -t hadoop_stop_daemon.bats
>  [exec] 1..2
>  [exec] ok 1 hadoop_stop_daemon_changing_pid
>  [exec] not ok 2 hadoop_stop_daemon_force_kill
>  [exec] # (in test file hadoop_stop_daemon.bats, line 43)
>  [exec] #   `[ -f ${TMP}/pidfile ]' failed
>  [exec] # bindir: 
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts
>  [exec] # sh: 
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts/process_with_sigterm_trap.sh:
>  No such file or directory{code}
> This happens because actually 3 commits belong to HADOOP-15527 but 
> HADOOP-15407 branch contains only one of them so the test won't find 
> process_with_sigterm_trap.sh.
> I am not sure what is the best practice to solve this kind of issues. Is 
> patch required or can somebody just cherry-pick the missing commits?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Reopened] (HADOOP-15668) mvn test goal fails on HADOOP-15407 branch

[Sunil Govindan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil Govindan reopened HADOOP-15668:
-

> mvn test goal fails on HADOOP-15407 branch
> --
>
> Key: HADOOP-15668
> URL: https://issues.apache.org/jira/browse/HADOOP-15668
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>
> It's very easy to reproduce:
> {code}cd hadoop-common-project/hadoop-common/
> mvn test -Dtest=Whatever{code}
> The error is due to
> {code} [exec] Running bats -t hadoop_stop_daemon.bats
>  [exec] 1..2
>  [exec] ok 1 hadoop_stop_daemon_changing_pid
>  [exec] not ok 2 hadoop_stop_daemon_force_kill
>  [exec] # (in test file hadoop_stop_daemon.bats, line 43)
>  [exec] #   `[ -f ${TMP}/pidfile ]' failed
>  [exec] # bindir: 
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts
>  [exec] # sh: 
> /Users/abokor/work/hadoop/hadoop-common-project/hadoop-common/src/test/scripts/process_with_sigterm_trap.sh:
>  No such file or directory{code}
> This happens because actually 3 commits belong to HADOOP-15527 but 
> HADOOP-15407 branch contains only one of them so the test won't find 
> process_with_sigterm_trap.sh.
> I am not sure what is the best practice to solve this kind of issues. Is 
> patch required or can somebody just cherry-pick the missing commits?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org