Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree [discussion -> lazy vote]

[Vinod Kumar Vavilapalli]

On Mon, Sep 23, 2019 at 4:02 PM Elek, Marton  wrote:

>  As the overall feedback was positive (in fact many of the answers were
> simple +1 votes) I don't think the thread should be repeated under
> [VOTE] subject. Therefore I call it for a lazy consensus.
>

Let's please not do this in the future.

These are large enough changes that we'd like to get formal votes both for
immediate visibility (VOTE threads are more in your face) as well as for
record-keeping in posterity.

Thanks
+Vinod

Re: VOTE PASSED - Hadoop-3.1.3-RC0 Re: [VOTE] Release Hadoop-3.1.3-RC0

[Vinod Kumar Vavilapalli]

Zhankun et.al,

Did we close down the rest of the release process for 3.1.3?

I don't yet see this on the site, so asking.

If it's not done yet and you are running into issues, please let me know
and I can help with the corresponding release management tasks.

Thanks
+Vinod

On Tue, Sep 24, 2019 at 2:16 PM Zhankun Tang  wrote:

> Hi,
>
> Thank you all who spent time to verify and vote 3.1.3 release!
> I‘m pleased to announce that the vote for 3.1.3 RC0 passed! I'll push the
> release bits and send out an announcement for 3.1.3 soon.
>
> Summary of votes for Hadoop-3.1.3-RC0:
>
> 4 binding +1s, from Rohith Sharma K S, Sunil Govindan, Weiwei Yang, Eric
> Payne
>
> 3 non-binding +1s, from Xun Liu, Zac Zhou, Zhankun Tang
>
> 1 non-binding with +0s from Masatake Iwasaki
>
> and *no -1s*.
>
> BR,
> Zhankun
>
> On Tue, 24 Sep 2019 at 15:40, zac yuan  wrote:
>
> > Looking forward to 3.1.3 release. Thanks Zhankun for your great effort~
> >
> > +1 (non-binding)
> >
> > Zac Zhou
> >
> > Xun Liu  于2019年9月23日周一 上午11:49写道：
> >
> > > +1
> > >
> > > Thanks Zhankun for all of your hard work on this release.
> > >
> > >
> > > > On Sep 20, 2019, at 5:34 AM, epa...@apache.org wrote:
> > > >
> > > >
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks Zhankun for all of your hard work on this release.
> > > >
> > > > I downloaded and built the source and ran it on an insecure
> multi-node
> > > pseudo cluster.
> > > >
> > > > I performed various YARN manual tests, including creating custom
> > > resources, creating queue submission ACLs, and queue refreshes.
> > > >
> > > > One concern is that preemption does not seem to be working when only
> > the
> > > custom resources are over the queue capacity, but I don't think this is
> > > something introduced with this release.
> > > >
> > > > -Eric
> > > >
> > > >
> > > >
> > > > On Thursday, September 12, 2019, 3:04:44 AM CDT, Zhankun Tang <
> > > zt...@apache.org> wrote:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Hi folks,
> > > >
> > > > Thanks to everyone's help on this release. Special thanks to Rohith,
> > > > Wei-Chiu, Akira, Sunil, Wangda!
> > > >
> > > > I have created a release candidate (RC0) for Apache Hadoop 3.1.3.
> > > >
> > > > The RC release artifacts are available at:
> > > > http://home.apache.org/~ztang/hadoop-3.1.3-RC0/
> > > >
> > > > The maven artifacts are staged at:
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1228/
> > > >
> > > > The RC tag in git is here:
> > > > https://github.com/apache/hadoop/tree/release-3.1.3-RC0
> > > >
> > > > And my public key is at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > *This vote will run for 7 days, ending on Sept.19th at 11:59 pm PST.*
> > > >
> > > > For the testing, I have run several Spark and distributed shell jobs
> in
> > > my
> > > > pseudo cluster.
> > > >
> > > > My +1 (non-binding) to start.
> > > >
> > > > BR,
> > > > Zhankun
> > > >
> > > > On Wed, 4 Sep 2019 at 15:56, zhankun tang 
> > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> Thanks for everyone helping in resolving all the blockers targeting
> > > Hadoop
> > > >> 3.1.3[1]. We've cleaned all the blockers and moved out non-blockers
> > > issues
> > > >> to 3.1.4.
> > > >>
> > > >> I'll cut the branch today and call a release vote soon. Thanks!
> > > >>
> > > >>
> > > >> [1]. https://s.apache.org/5hj5i
> > > >>
> > > >> BR,
> > > >> Zhankun
> > > >>
> > > >>
> > > >> On Wed, 21 Aug 2019 at 12:38, Zhankun Tang 
> wrote:
> > > >>
> > > >>> Hi folks,
> > > >>>
> > > >>> We have Apache Hadoop 3.1.2 released on Feb 2019.
> > > >>>
> > > >>> It's been more than 6 months passed and there're
> > > >>>
> > > >>> 246 fixes[1]. 2 blocker and 4 critical Issues [2]
> > > >>>
> > > >>> (As Wei-Chiu Chuang mentioned, HDFS-13596 will be another blocker)
> > > >>>
> > > >>>
> > > >>> I propose my plan to do a maintenance release of 3.1.3 in the next
> > few
> > > >>> (one or two) weeks.
> > > >>>
> > > >>> Hadoop 3.1.3 release plan:
> > > >>>
> > > >>> Code Freezing Date: *25th August 2019 PDT*
> > > >>>
> > > >>> Release Date: *31th August 2019 PDT*
> > > >>>
> > > >>>
> > > >>> Please feel free to share your insights on this. Thanks!
> > > >>>
> > > >>>
> > > >>> [1] https://s.apache.org/zw8l5
> > > >>>
> > > >>> [2] https://s.apache.org/fjol5
> > > >>>
> > > >>>
> > > >>> BR,
> > > >>>
> > > >>> Zhankun
> > > >>>
> > > >>
> > > >
> > > > -
> > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: Does VOTE necessary to create a child repo?

[Vinod Kumar Vavilapalli]

Moving the thread to the dev lists.

Thanks
+Vinod

> On Sep 23, 2019, at 11:43 PM, Vinayakumar B  wrote:
> 
> Thanks Marton,
> 
> Current created 'hadoop-thirdparty' repo is empty right now.
> Whether to use that repo  for shaded artifact or not will be monitored in
> HADOOP-13363 umbrella jira. Please feel free to join the discussion.
> 
> There is no existing codebase is being moved out of hadoop repo. So I think
> right now we are good to go.
> 
> -Vinay
> 
> On Mon, Sep 23, 2019 at 11:38 PM Marton Elek  wrote:
> 
>> 
>> I am not sure if it's defined when is a vote required.
>> 
>> https://www.apache.org/foundation/voting.html
>> 
>> Personally I think it's a big enough change to send a notification to the
>> dev lists with a 'lazy consensus'  closure
>> 
>> Marton
>> 
>> On 2019/09/23 17:46:37, Vinayakumar B  wrote:
>>> Hi,
>>> 
>>> As discussed in HADOOP-13363, protobuf 3.x jar (and may be more in
>> future)
>>> will be kept as a shaded artifact in a separate repo, which will be
>>> referred as dependency in hadoop modules.  This approach avoids shading
>> of
>>> every submodule during build.
>>> 
>>> So question is does any VOTE required before asking to create a git repo?
>>> 
>>> On selfserve platform https://gitbox.apache.org/setup/newrepo.html
>>> I can access see that, requester should be PMC.
>>> 
>>> Wanted to confirm here first.
>>> 
>>> -Vinay
>>> 
>> 
>> -
>> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: private-h...@hadoop.apache.org
>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] Apache Hadoop 3.2.1 release

[Vinod Kumar Vavilapalli]

Done: https://twitter.com/hadoop/status/1176787511865008128.

If you have tweetdeck, any of the PMC members can do this.

BTW, it looks we haven't published any releases since Nov 2018. Let's get back 
to doing this going forward!

Thanks
+Vinod

> On Sep 25, 2019, at 2:44 PM, Rohith Sharma K S  
> wrote:
> 
> Updated twitter message:
> 
> ``
> Apache Hadoop 3.2.1 is released: https://s.apache.org/96r4h
> 
> Announcement: https://s.apache.org/jhnpe
> Overview: https://s.apache.org/tht6a
> Changes: https://s.apache.org/pd6of
> Release notes: https://s.apache.org/ta50b
> 
> Thanks to our community of developers, operators, and users.
> 
> 
> -Rohith Sharma K S
> 
> 
> On Wed, 25 Sep 2019 at 14:15, Sunil Govindan  wrote:
> 
>> Here the link of Overview URL is old.
>> We should ideally use https://hadoop.apache.org/release/3.2.1.html
>> 
>> Thanks
>> Sunil
>> 
>> On Wed, Sep 25, 2019 at 2:10 PM Rohith Sharma K S <
>> rohithsharm...@apache.org> wrote:
>> 
>>> Can someone help to post this in twitter account?
>>> 
>>> Apache Hadoop 3.2.1 is released: https://s.apache.org/mzdb6
>>> Overview: https://s.apache.org/tht6a
>>> Changes: https://s.apache.org/pd6of
>>> Release notes: https://s.apache.org/ta50b
>>> 
>>> Thanks to our community of developers, operators, and users.
>>> 
>>> -Rohith Sharma K S
>>> 
>>> On Wed, 25 Sep 2019 at 13:44, Rohith Sharma K S <
>>> rohithsharm...@apache.org> wrote:
>>> 
 Hi all,
 
It gives us great pleasure to announce that the Apache Hadoop
 community has
 voted to release Apache Hadoop 3.2.1.
 
 Apache Hadoop 3.2.1 is the stable release of Apache Hadoop 3.2 line,
 which
 includes 493 fixes since Hadoop 3.2.0 release:
 
 - For major changes included in Hadoop 3.2 line, please refer Hadoop
 3.2.1 main page[1].
 - For more details about fixes in 3.2.1 release, please read
 CHANGELOG[2] and RELEASENOTES[3].
 
 The release news is posted on the Hadoop website too, you can go to the
 downloads section directly[4].
 
 Thank you all for contributing to the Apache Hadoop!
 
 Cheers,
 Rohith Sharma K S
 
 
 [1] https://hadoop.apache.org/docs/r3.2.1/index.html
 [2]
 https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-common/release/3.2.1/CHANGELOG.3.2.1.html
 [3]
 https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-common/release/3.2.1/RELEASENOTES.3.2.1.html
 [4] https://hadoop.apache.org
 
>>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [NOTICE] Building trunk needs protoc 3.7.1

[Vinod Kumar Vavilapalli]

Quick question, being lazy here, lots of JIRA updates on HADOOP-13363 over the 
years not helping either.

Does anyone know what this upgrade will mean w.r.t compatibility for the Hadoop 
releases themselves? Remember that trunk is still 3.x.

Thanks
+Vinod

> On Sep 21, 2019, at 9:55 AM, Vinayakumar B  wrote:
> 
> @Wangda Tan  ,
> Sorry for the confusion. HADOOP-13363 is umbrella jira to track multiple
> stages of protobuf upgrade in subtasks. (jar upgrade, Docker update, plugin
> upgrade, shading, etc).
> Right now, first task of jar upgrade is done. So need to update the protoc
> executable in the in build environments.
> 
> @张铎(Duo Zhang)  ,
> Sorry for the inconvenience. Yes, indeed plugin update before jar upgrde
> was possible. Sorry I missed it.
> 
> Plugin update needed to be done for whole project, for which precommit
> jenkins will need more time complete end-to-end runs.
> So plugin update is planned in stages in further subtasks. It could be done
> in 2-3 days.
> 
> -Vinay
> 
> On Sat, 21 Sep 2019, 5:55 am 张铎(Duo Zhang),  wrote:
> 
>> I think this one is alread in place so we have to upgrade...
>> 
>> https://issues.apache.org/jira/browse/HADOOP-16557
>> 
>> Wangda Tan  于2019年9月21日周六 上午7:19写道：
>> 
>>> Hi Vinay,
>>> 
>>> A bit confused, I saw the HADOOP-13363 is still pending. Do we need to
>>> upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is completed?
>>> 
>>> Thanks,
>>> Wangda
>>> 
>>> On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B 
>>> wrote:
>>> 
 Hi All,
 
 A very long pending task, protobuf upgrade is happening in
>> HADOOP-13363.
>>> As
 part of that protobuf version is upgraded to 3.7.1.
 
 Please update your build environments to have 3.7.1 protobuf version.
 
 BUILIDING.txt has been updated with latest instructions.
 
 This pre-requisite to update protoc dependecy manually is required
>> until
 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to
 dynamically resolve required protoc exe.
 
 Dockerfile is being updated to have latest 3.7.1 as default protoc for
>>> test
 environments.
 
 Thanks,
 -Vinay
 
>>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree

[Vinod Kumar Vavilapalli]

Looks to me that the advantages of this additional step are only incremental 
given that you've already decoupled releases and dependencies.

Do you see a Submarine like split-also-into-a-TLP for Ozone? If not now, 
sometime further down the line? If so, why not do both at the same time? I felt 
the same way with Submarine, but couldn't follow up in time.

Thanks
+Vinod

> On Sep 18, 2019, at 4:04 AM, Wangda Tan  wrote:
> 
> +1 (binding).
> 
> From my experiences of Submarine project, I think moving to a separate repo
> helps.
> 
> - Wangda
> 
> On Tue, Sep 17, 2019 at 11:41 AM Subru Krishnan  wrote:
> 
>> +1 (binding).
>> 
>> IIUC, there will not be an Ozone module in trunk anymore as that was my
>> only concern from the original discussion thread? IMHO, this should be the
>> default approach for new modules.
>> 
>> On Tue, Sep 17, 2019 at 9:58 AM Salvatore LaMendola (BLOOMBERG/ 731 LEX) <
>> slamendo...@bloomberg.net> wrote:
>> 
>>> +1
>>> 
>>> From: e...@apache.org At: 09/17/19 05:48:32To:
>> hdfs-...@hadoop.apache.org,
>>> mapreduce-...@hadoop.apache.org,  common-...@hadoop.apache.org,
>>> yarn-dev@hadoop.apache.org
>>> Subject: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk
>>> source tree
>>> 
>>> 
>>> TLDR; I propose to move Ozone related code out from Hadoop trunk and
>>> store it in a separated *Hadoop* git repository apache/hadoop-ozone.git
>>> 
>>> 
>>> When Ozone was adopted as a new Hadoop subproject it was proposed[1] to
>>> be part of the source tree but with separated release cadence, mainly
>>> because it had the hadoop-trunk/SNAPSHOT as compile time dependency.
>>> 
>>> During the last Ozone releases this dependency is removed to provide
>>> more stable releases. Instead of using the latest trunk/SNAPSHOT build
>>> from Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now).
>>> 
>>> As we have no more strict dependency between Hadoop trunk SNAPSHOT and
>>> Ozone trunk I propose to separate the two code base from each other with
>>> creating a new Hadoop git repository (apache/hadoop-ozone.git):
>>> 
>>> With moving Ozone to a separated git repository:
>>> 
>>>  * It would be easier to contribute and understand the build (as of now
>>> we always need `-f pom.ozone.xml` as a Maven parameter)
>>>  * It would be possible to adjust build process without breaking
>>> Hadoop/Ozone builds.
>>>  * It would be possible to use different Readme/.asf.yaml/github
>>> template for the Hadoop Ozone and core Hadoop. (For example the current
>>> github template [2] has a link to the contribution guideline [3]. Ozone
>>> has an extended version [4] from this guideline with additional
>>> information.)
>>>  * Testing would be more safe as it won't be possible to change core
>>> Hadoop and Hadoop Ozone in the same patch.
>>>  * It would be easier to cut branches for Hadoop releases (based on the
>>> original consensus, Ozone should be removed from all the release
>>> branches after creating relase branches from trunk)
>>> 
>>> 
>>> What do you think?
>>> 
>>> Thanks,
>>> Marton
>>> 
>>> [1]:
>>> 
>>> 
>> https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8
>>> c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E
>>> [2]:
>>> 
>>> 
>> https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md
>>> [3]:
>> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
>>> [4]:
>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone
>>> 
>>> -
>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>> 
>>> 
>>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Any thoughts making Submarine a separate Apache project?

[Vinod Kumar Vavilapalli]

Looks like there's a meaningful push behind this.

Given the desire is to fork off Apache Hadoop, you'd want to make sure this 
enthusiasm turns into building a real, independent but more importantly a 
sustainable community.

Given that there were two official releases off the Apache Hadoop project, I 
doubt if you'd need to go through the incubator process. Instead you can 
directly propose a new TLP at ASF board. The last few times this happened was 
with ORC, and long before that with Hive, HBase etc. Can somebody who have 
cycles and been on the ASF lists for a while look into the process here?

For the Apache Hadoop community, this will be treated simply as code-change and 
so need a committer +1? You can be more gently by formally doing a vote once a 
process doc is written down.

Back to the sustainable community point, as part of drafting this proposal, 
you'd definitely want to make sure all of the Apache Hadoop PMC/Committers can 
exercise their will to join this new project as PMC/Committers respectively 
without any additional constraints.

Thanks
+Vinod

> On Jul 25, 2019, at 1:31 PM, Wangda Tan  wrote:
> 
> Thanks everybody for sharing your thoughts. I saw positive feedbacks from
> 20+ contributors!
> 
> So I think we should move it forward, any suggestions about what we should
> do?
> 
> Best,
> Wangda
> 
> On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:
> 
>> +1, This is neo from TiDB & TiKV community.
>> Thanks Xun for bring this up.
>> 
>> Our CNCF project's open source distributed KV storage system TiKV,
>> Hadoop submarine's machine learning engine helps us to optimize data
>> storage,
>> helping us solve some problems in data hotspots and data shuffers.
>> 
>> We are ready to improve the performance of TiDB in our open source
>> distributed relational database TiDB and also using the hadoop submarine
>> machine learning engine.
>> 
>> I think if submarine can be independent, it will develop faster and better.
>> Thanks to the hadoop community for developing submarine!
>> 
>> Best Regards,
>> neo
>> www.pingcap.com / https://github.com/pingcap/tidb /
>> https://github.com/tikv
>> 
>> Xun Liu  于2019年7月22日周一 下午4:07写道：
>> 
>>> @adam.antal
>>> 
>>> The submarine development team has completed the following preparations:
>>> 1. Established a temporary test repository on Github.
>>> 2. Change the package name of hadoop submarine from org.hadoop.submarine
>> to
>>> org.submarine
>>> 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
>>> 4. On the Github docked travis-ci system, all test cases have been
>> tested;
>>> 5. Several Hadoop submarine users completed the system test using the
>> code
>>> in this repository.
>>> 
>>> 赵欣  于2019年7月22日周一 上午9:38写道：
>>> 
 Hi
 
 I am a teacher at Southeast University (https://www.seu.edu.cn/). We
>> are
 a major in electrical engineering. Our teaching teams and students use
 bigoop submarine for big data analysis and automation control of
>>> electrical
 equipment.
 
 Many thanks to the hadoop community for providing us with machine
>>> learning
 tools like submarine.
 
 I wish hadoop submarine is getting better and better.
 
 
 ==
 赵欣
 东南大学电气工程学院
 
 -
 
 Zhao XIN
 
 School of Electrical Engineering
 
 ==
 2019-07-18
 
 
 *From:* Xun Liu 
 *Date:* 2019-07-18 09:46
 *To:* xinzhao 
 *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
 project?
 
 
 -- Forwarded message -
 发件人： dashuiguailu...@gmail.com 
 Date: 2019年7月17日周三 下午3:17
 Subject: Re: Re: Any thoughts making Submarine a separate Apache
>> project?
 To: Szilard Nemeth , runlin zhang <
 runlin...@gmail.com>
 Cc: Xun Liu , common-dev <
>>> common-...@hadoop.apache.org>,
 yarn-dev , hdfs-dev <
 hdfs-...@hadoop.apache.org>, mapreduce-dev <
 mapreduce-...@hadoop.apache.org>, submarine-dev <
 submarine-...@hadoop.apache.org>
 
 
 +1 ，Good idea, we are very much looking forward to it.
 
 --
 dashuiguailu...@gmail.com
 
 
 *From:* Szilard Nemeth 
 *Date:* 2019-07-17 14:55
 *To:* runlin zhang 
 *CC:* Xun Liu ; Hadoop Common
 ; yarn-dev ;
 Hdfs-dev ; mapreduce-dev
 ; submarine-dev
 
 *Subject:* Re: Any thoughts making Submarine a separate Apache project?
 +1, this is a very great idea.
 As Hadoop repository has already grown huge and contains many
>> projects, I
 think in general it's a good idea to separate projects in the early
>>> phase.
 
 
 On Wed, Jul 17, 2019, 08:50 runlin zhang  wrote:
 
> +1 ，That will be great ！
> 
>> 在 2019年7月10日，下午3:34，Xun Liu  写道：
>> 
>> Hi all,
>> 
>> This is Xun Liu contributing to the Submarine 

Re: [DISCUSS] Prefer JUnit5 for new tests

[Vinod Kumar Vavilapalli]

+1 if it is indeed possible to have both.

Thanks
+Vinod

> On Jul 26, 2019, at 1:56 PM, Akira Ajisaka  wrote:
> 
> Hi folks,
> 
> Now we are slowly migrating from JUnit4 to JUnit5.
> https://issues.apache.org/jira/browse/HADOOP-14693
> 
> However, as Steve commented [1], if we are going to migrate the
> existing tests, the backporting cost will become too expensive.
> Therefore, I'd like to recommend using JUnit5 for new tests before
> migrating the existing tests. Using junit-vintage-engine, we can mix
> JUnit4 and JUnit5 APIs in the same module, so writing new tests in
> JUnit5 is relatively easy.
> 
> Any thoughts?
> 
> [1] 
> https://issues.apache.org/jira/browse/HADOOP-16318?focusedCommentId=16890955=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16890955
> 
> -Akira
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Force "squash and merge" option for PR merge on github UI

[Vinod Kumar Vavilapalli]

Makes sense, +1.

Thanks
+Vinod

> On Jul 17, 2019, at 11:37 AM, Elek, Marton  wrote:
> 
> Hi,
> 
> Github UI (ui!) helps to merge Pull Requests to the proposed branch.
> There are three different ways to do it [1]:
> 
> 1. Keep all the different commits from the PR branch and create one
> additional merge commit ("Create a merge commit")
> 
> 2. Squash all the commits and commit the change as one patch ("Squash
> and merge")
> 
> 3. Keep all the different commits from the PR branch but rebase, merge
> commit will be missing ("Rebase and merge")
> 
> 
> 
> As only the option 2 is compatible with the existing development
> practices of Hadoop (1 issue = 1 patch = 1 commit), I call for a lazy
> consensus vote: If no objections withing 3 days, I will ask INFRA to
> disable the options 1 and 3 to make the process less error prone.
> 
> Please let me know, what do you think,
> 
> Thanks a lot
> Marton
> 
> ps: Personally I prefer to merge from local as it enables to sign the
> commits and do a final build before push. But this is a different story,
> this proposal is only about removing the options which are obviously
> risky...
> 
> ps2: You can always do any kind of merge / commits from CLI, for example
> to merge a feature branch together with keeping the history.
> 
> [1]:
> https://help.github.com/en/articles/merging-a-pull-request#merging-a-pull-request-on-github
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Fwd: Incorrect NOTICE files in TLP releases

[Vinod Kumar Vavilapalli]

A bit of an old email, but want to make sure this isn't missed.

Has anyone looked into this concern?

Ref https://issues.apache.org/jira/browse/ROL-2138 
.

Thanks
+Vinod

> Begin forwarded message:
> 
> From: sebb 
> Subject: Incorrect NOTICE files in TLP releases
> Date: June 21, 2019 at 2:34:17 AM GMT+5:30
> To: "bo...@apache.org Board" 
> Reply-To: bo...@apache.org
> 
> To whom it may concern:
> 
> I had occasion to download the source for Roller, and happened to look
> at the NOTICE file.
> It does not conform to ASF policies, so I raised ROL-2138.
> 
> One of the replies asked how to manage different N files for binary
> and source releases, and pointed out that Hadoop and Karaf don't
> appear to have multiple copies of the files.
> 
> So I had a look at Hadoop and Karaf; their NOTICE files are also
> non-standard, and it looks like Kafka does not have a NOTICE file in
> the source bundle.
> 
> I suspect these are not the only projects with non-conformant NOTICE files.
> The LICENSE files are also likely to be incorrect (I have not checked).
> 
> Sebb.

Re: June Hadoop Community Meetup

[Vinod Kumar Vavilapalli]

Plus general@ list.

Thanks
+Vinod

> On Jun 4, 2019, at 9:47 PM, Daniel Templeton  
> wrote:
> 
> The meetup page is now live:
> 
>https://www.meetup.com/Hadoop-Contributors/events/262055924
> 
> I'll fill in the agenda details after we get them nailed down.  The meetup 
> will be an all-day event on June 26th with lunch provided and a reception 
> after.  Let me know if there are any questions.
> 
> Hope to see you there!
> Daniel
> 
> On 5/23/19 10:57 AM, Daniel Templeton wrote:
>> Hi, all!  I want to let you know that Cloudera is planning to host a 
>> contributors meetup on June 26 at our Palo Alto headquarters. We're still 
>> working out the details, but we're hoping to follow the format that Oath and 
>> LinkedIn followed during the last two. Please feel free to reach out to me 
>> if you have a topic you'd like to propose for the meetup.  I will also be 
>> reaching out to key folks from the community to solicit ideas.  I will send 
>> out an update with more details when I have more to share.
>> 
>> Thanks!
>> Daniel
> 
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Impact of Not Enabling KerberosDelegationTokenAuthenticationHandler

[Vinod Kumar Vavilapalli]

Not sure what your question really is.

KerberosDelegationTokenAuthenticationHandler was created mainly to add 
delegation-token support on top of KerberosAuthenticationHandler.

Thanks
+Vinod

> On May 22, 2019, at 10:58 AM, Prabhu Joseph  
> wrote:
> 
> Hi,
> 
> Was testing YARN RM with Http AuthenticationFilter set with
> KerberosAuthenticationHandler which authenticates only through Kerberos and
> not Delegation Token. Have observed everything like Yarn jobs, yarn
> commands, UI and Rest API working fine with this setup with valid kerberos
> ticket.
> 
> Want to understand where the impact is then when not using
> KerberosDelegationTokenAuthenticationHandler.
> 
> Thanks,
> Prabhu Joseph


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Cluster Submit Applications API accepts any random ApplicationId

[Vinod Kumar Vavilapalli]

This is an old but known issue. I tried to search for JIRAs where we discussed 
this but couldn't easily.

We have so far only dealt with mistake by well behaved clients, not bad 
users/clients. YARN-1545 was filed for some of this, but never reached fruition.

It will definitely have an impact if a malicious user does this. You should be 
careful of the restart scenario where a client gets the application-id from the 
old RM and submits the app to the new RM. If you decide to fix this, you will 
need to make changes on YarnClient also to address that case.

Thanks
+Vinod

> On May 20, 2019, at 2:31 AM, Prabhu Joseph  wrote:
> 
> Hi,
> 
>  Have observed YARN Cluster Submit Applications API accepts any random 
> ApplicationId which is not provided by Cluster New Application API. There is 
> no enforcer to check if the ApplicationId is provided by RM.  User can pass 
> applicationId with different clusterTimestamp, negative clusterTimestamp, 
> negative Id. Not sure if this will have any impact. But as per the doc, 
> ApplicationId must be obtained from New Application API. 
> 
> Cluster Applications API(Submit Application)
> 
> The Submit Applications API can be used to submit applications. In case of 
> submitting applications, you must first obtain an application-id using the 
> Cluster New Application API 
> .
> 
> 
> Want to check if using random ApplicationIs is an expected behavior and won't 
> have any impact.
> 
> Thanks,
> Prabhu Joseph
> 
> 

[jira] [Created] (YARN-9548) [Umbrella] Make YARN work well in elastic cloud environments

[Vinod Kumar Vavilapalli (JIRA)]

Vinod Kumar Vavilapalli created YARN-9548:
-

 Summary: [Umbrella] Make YARN work well in elastic cloud 
environments
 Key: YARN-9548
 URL: https://issues.apache.org/jira/browse/YARN-9548
 Project: Hadoop YARN
  Issue Type: New Feature
Reporter: Vinod Kumar Vavilapalli


YARN works well in static environments but there isn't fundamentally broken in 
YARN to stop us from making it work well in dynamic environments like cloud 
(public or private) as well.

There are few areas where we need to invest though
 # Autoscaling
 -- cluster level: add/remove nodes intelligently based on metrics and/or admin 
plugins
 -- node level: scale nodes up/down vertically?
 # Smarter scheduling
-- to pack containers as opposed to spreading them around to account for 
nodes going away
-- to account for speculative nodes like spot instances
 # Handling nodes going away better
-- by decommissioning sanely
-- dealing with auxiliary services data
 # And any installation helpers in this dynamic world - scripts, operators etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Cannot kill Pre-Commit jenkins builds

[Vinod Kumar Vavilapalli]

Done. Please see if it works for you.

Thanks
+Vinod

> On Feb 14, 2019, at 1:12 PM, Arun Suresh  wrote:
> 
> Hello Vinod
> 
> Kindly add me (asuresh) and Jonathan Hung as well..
> 
> Cheers
> -Arun
> 
> On Thu, Feb 14, 2019, 11:46 AM Vinod Kumar Vavilapalli  <mailto:vino...@apache.org>> wrote:
> Plus Sunil / Wangda.
> 
> Sunil found out some information - "Jenkins uses the Apache LDAP servers for 
> authentication. To give a committer access to Jenkins, the committer must be 
> made a member of the hudson-jobadmin group. This is done using the Whimsy 
> Tool which all PMC Chairs can change."
> 
> I've done this for Sunil & Wangda as they needed help for Submarine.
> 
> Arun Suresh and other release-managers / committers, let me know if you need 
> to be added.
> 
> Thanks
> +Vinod
> 
> > On Jan 22, 2019, at 4:50 PM, Arun Suresh  > <mailto:arun.sur...@gmail.com>> wrote:
> > 
> > Hmmm.. as per this (https://wiki.apache.org/general/Jenkins 
> > <https://wiki.apache.org/general/Jenkins>), looks like my
> > id needs to be added to the hudson-jobadmin group to affect any changes on
> > jenkins.
> > But wondering why it was revoked in the first place.
> > 
> > On Tue, Jan 22, 2019 at 4:21 PM Vinod Kumar Vavilapalli  > <mailto:vino...@apache.org>>
> > wrote:
> > 
> >> Minus private.
> >> 
> >> Which specific job you are looking? I looked at
> >> https://builds.apache.org/job/PreCommit-YARN-Build/ 
> >> <https://builds.apache.org/job/PreCommit-YARN-Build/> but can't seem to
> >> find any user specific auth.
> >> 
> >> +Vinod
> >> 
> >> On Jan 22, 2019, at 10:00 AM, Arun Suresh  >> <mailto:asur...@apache.org>> wrote:
> >> 
> >> Hey Vinod.. Ping!
> >> 
> >> Cheers
> >> -Arun
> >> 
> >> On Fri, Jan 18, 2019 at 9:46 AM Arun Suresh  >> <mailto:asur...@apache.org>> wrote:
> >> 
> >> Hi Vinod
> >> 
> >> Can you please help with this:
> >> https://issues.apache.org/jira/browse/INFRA-17673 
> >> <https://issues.apache.org/jira/browse/INFRA-17673> ?
> >> 
> >> Cheers
> >> -Arun
> >> 
> >> On Wed, Jan 16, 2019, 12:53 PM Arun Suresh  >> <mailto:asur...@apache.org> wrote:
> >> 
> >> Hi
> >> 
> >> We are currently trying to get the branch-2 pre-commit builds working.
> >> I used to be able to kill Pre-Commit jenkins jobs, but looks like I am
> >> not allowed to anymore. Has anything changed recently w.r.t permissions etc
> >> ?
> >> 
> >> Cheers
> >> -Arun
> >> 
> >> 
> >> 
> >> 
> 

Re: Cannot kill Pre-Commit jenkins builds

[Vinod Kumar Vavilapalli]

Plus Sunil / Wangda.

Sunil found out some information - "Jenkins uses the Apache LDAP servers for 
authentication. To give a committer access to Jenkins, the committer must be 
made a member of the hudson-jobadmin group. This is done using the Whimsy Tool 
which all PMC Chairs can change."

I've done this for Sunil & Wangda as they needed help for Submarine.

Arun Suresh and other release-managers / committers, let me know if you need to 
be added.

Thanks
+Vinod

> On Jan 22, 2019, at 4:50 PM, Arun Suresh  wrote:
> 
> Hmmm.. as per this (https://wiki.apache.org/general/Jenkins), looks like my
> id needs to be added to the hudson-jobadmin group to affect any changes on
> jenkins.
> But wondering why it was revoked in the first place.
> 
> On Tue, Jan 22, 2019 at 4:21 PM Vinod Kumar Vavilapalli 
> wrote:
> 
>> Minus private.
>> 
>> Which specific job you are looking? I looked at
>> https://builds.apache.org/job/PreCommit-YARN-Build/ but can't seem to
>> find any user specific auth.
>> 
>> +Vinod
>> 
>> On Jan 22, 2019, at 10:00 AM, Arun Suresh  wrote:
>> 
>> Hey Vinod.. Ping!
>> 
>> Cheers
>> -Arun
>> 
>> On Fri, Jan 18, 2019 at 9:46 AM Arun Suresh  wrote:
>> 
>> Hi Vinod
>> 
>> Can you please help with this:
>> https://issues.apache.org/jira/browse/INFRA-17673 ?
>> 
>> Cheers
>> -Arun
>> 
>> On Wed, Jan 16, 2019, 12:53 PM Arun Suresh > 
>> Hi
>> 
>> We are currently trying to get the branch-2 pre-commit builds working.
>> I used to be able to kill Pre-Commit jenkins jobs, but looks like I am
>> not allowed to anymore. Has anything changed recently w.r.t permissions etc
>> ?
>> 
>> Cheers
>> -Arun
>> 
>> 
>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Moving branch-2 to java 8

[Vinod Kumar Vavilapalli]

The community made a decision long time ago that we'd like to keep the 
compatibility & so tie branch-2 to Java 7, but do Java 8+ only work on 3.x.

I always assumed that most (all?) downstream users build branch-2 on JDK 7 
only, can anyone confirm? If so, there may be an easier way to address these 
test issues.

+Vinod

> On Jan 28, 2019, at 11:24 AM, Jonathan Hung  wrote:
> 
> Hi folks,
> 
> Forking a discussion based on HADOOP-15711. To summarize, there are issues
> with branch-2 tests running on java 7 (openjdk) which don't exist on java
> 8. From our testing, the build can pass with openjdk 8.
> 
> For branch-3, the work to move the build to use java 8 was done in
> HADOOP-14816 as part of the Dockerfile OS version change. HADOOP-16053 was
> filed to backport this OS version change to branch-2 (but without the java
> 7 -> java 8 change). So my proposal is to also make the java 7 -> java 8
> version change in branch-2.
> 
> As mentioned in HADOOP-15711, the main issue is around source and binary
> compatibility. I don't currently have a great answer, but one initial
> thought is to build source/binary against java 7 to ensure compatibility
> and run the rest of the build as java 8.
> 
> Thoughts?
> 
> Jonathan Hung


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Cannot kill Pre-Commit jenkins builds

[Vinod Kumar Vavilapalli]

Minus private.

Which specific job you are looking? I looked at 
https://builds.apache.org/job/PreCommit-YARN-Build/ 
 but can't seem to find 
any user specific auth.

+Vinod

> On Jan 22, 2019, at 10:00 AM, Arun Suresh  wrote:
> 
> Hey Vinod.. Ping!
> 
> Cheers
> -Arun
> 
> On Fri, Jan 18, 2019 at 9:46 AM Arun Suresh  wrote:
> 
>> Hi Vinod
>> 
>> Can you please help with this:
>> https://issues.apache.org/jira/browse/INFRA-17673 ?
>> 
>> Cheers
>> -Arun
>> 
>> On Wed, Jan 16, 2019, 12:53 PM Arun Suresh > 
>>> Hi
>>> 
>>> We are currently trying to get the branch-2 pre-commit builds working.
>>> I used to be able to kill Pre-Commit jenkins jobs, but looks like I am
>>> not allowed to anymore. Has anything changed recently w.r.t permissions etc
>>> ?
>>> 
>>> Cheers
>>> -Arun
>>> 
>> 

Re: [Result] [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

[Vinod Kumar Vavilapalli]

Agree, it isn't productive this way.

I can't seem to find it, but was there a DISCUSS thread for this branch-merge? 
I usually recommend addressing issues on a DISCUSS thread instead of fighting 
things over a VOTE.

+Vinod

> On Dec 13, 2018, at 10:09 AM, Konstantin Shvachko  
> wrote:
> 
> This vote failed due to Daryn Sharp's veto.
> The concern is being addressed by HDFS-13873. I will start a new vote once
> this is committed.
> 
> Note for Daryn. Your non-responsive handling of the veto makes a bad
> precedence and is a bad example of communication on the lists from a
> respected member of this community. Please check your availability for
> followup discussions if you choose to get involved with important decisions.
> 
> On Fri, Dec 7, 2018 at 4:10 PM Konstantin Shvachko 
> wrote:
> 
>> Hi Daryn,
>> 
>> Wanted to backup Chen's earlier response to your concerns about rotating
>> calls in the call queue.
>> Our design
>> 1. targets directly the livelock problem by rejecting calls on the
>> Observer that are not likely to be responded in timely matter: HDFS-13873.
>> 2. The call queue rotation is only done on Observers, and never on the
>> active NN, so it stays free of attacks like you suggest.
>> 
>> If this is a satisfactory mitigation for the problem could you please
>> reconsider your -1, so that people could continue voting on this thread.
>> 
>> Thanks,
>> --Konst
>> 
>> On Thu, Dec 6, 2018 at 10:38 AM Daryn Sharp  wrote:
>> 
>>> -1 pending additional info.  After a cursory scan, I have serious
>>> concerns regarding the design.  This seems like a feature that should have
>>> been purely implemented in hdfs w/o touching the common IPC layer.
>>> 
>>> The biggest issue in the alignment context.  It's purpose appears to be
>>> for allowing handlers to reinsert calls back into the call queue.  That's
>>> completely unacceptable.  A buggy or malicious client can easily cause
>>> livelock in the IPC layer with handlers only looping on calls that never
>>> satisfy the condition.  Why is this not implemented via RetriableExceptions?
>>> 
>>> On Thu, Dec 6, 2018 at 1:24 AM Yongjun Zhang 
>>> wrote:
>>> 
 Great work guys.
 
 Wonder if we can elaborate what's impact of not having #2 fixed, and why
 #2
 is not needed for the feature to complete?
 2. Need to fix automatic failover with ZKFC. Currently it does not
 doesn't
 know about ObserverNodes trying to convert them to SBNs.
 
 Thanks.
 --Yongjun
 
 
 On Wed, Dec 5, 2018 at 5:27 PM Konstantin Shvachko  
 wrote:
 
> Hi Hadoop developers,
> 
> I would like to propose to merge to trunk the feature branch
 HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale
 read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC
 workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
> 
> The main functionality has been implemented see sub-tasks of
 HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
> 
> There are a few outstanding issues:
> 1. Need to provide proper documentation - a user guide for the new
 feature
> 2. Need to fix automatic failover with ZKFC. Currently it does not
 doesn't
> know about ObserverNodes trying to convert them to SBNs.
> 3. Scale testing and performance fine-tuning
> 4. As testing progresses, we continue fixing non-critical bugs like
> HDFS-14116.
> 
> I attached a unified patch to the umbrella jira for the review and
 Jenkins
> build.
> Please vote on this thread. The vote will run for 7 days until Wed Dec
 12.
> 
> Thanks,
> --Konstantin
> 
 
>>> 
>>> 
>>> --
>>> 
>>> Daryn
>>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Move to gitbox

[Vinod Kumar Vavilapalli]

We need to write up the stages of the process and put a concrete timeline first.

Is it an atomic move? Or will both repos live at the same time? Do we need a 
stop-the-world period and messaging sent out?

When do the git-wip repos get deleted and/or stop getting updates?

Things I can think of that need tracking
 - Wiki pages?
 - Jenkins jobs
 - Email notification setup for the commits?
 - Policies - like no-mainline-branch-deletion support

+Vinod

> On Dec 13, 2018, at 7:57 AM, Elek, Marton  wrote:
> 
> 
> 
> On 12/12/18 12:27 PM, Akira Ajisaka wrote:
>> Thank you for your positive feedback! I'll file a jira to INFRA in this 
>> weekend.
>> 
>>> If I understood well the only bigger task here is to update all the jenkins 
>>> jobs. (I am happy to help/contribute what I can do)
> 
>> Thank you Elek for the information. Do you have the privilege to
>> update the Jenkins jobs?
>> 
> I have, but I am more familiar with the Ozone jenkins jobs. I created a
> jira (HADOOP-16003) to discuss / record the changes / where it can be
> discussed or commented by anybody with more expertise.
> 
> Marton
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.1.2 release plan

[Vinod Kumar Vavilapalli]

231 fixed JIRAs is already quite a bunch!

I only see 7 JIRAs marked with Affects Version 3.1.2 and only one of them as 
blocker.

Why not just release now as soon as there are no blockers?

Thanks
+Vinod

> On Oct 24, 2018, at 4:36 PM, Wangda Tan  wrote:
> 
> Hi, All
> 
> We have released Apache Hadoop 3.1.1 on Aug 8, 2018. To further
> improve the quality of the release, I plan to release 3.1.2
> by Nov. The focus of 3.1.2 will be fixing blockers / critical bugs
> and other enhancements. So far there are 231 JIRAs [1] have fix
> version marked to 3.1.2
> 
> I plan to cut branch-3.1 on Nov 15 and vote for RC on the same day.
> 
> Please feel free to share your insights.
> 
> Thanks,
> Wangda Tan
> 
> [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce")
> AND fixVersion = 3.1.2


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Vinod Kumar Vavilapalli]

Is there no way to host the new site and the old site concurrently? And link 
back & forth?

+Vinod


> On Aug 31, 2018, at 1:07 AM, Elek, Marton  wrote:
> 
> Bumping this thread at last time.
> 
> I have the following proposal:
> 
> 1. I will request a new git repository hadoop-site.git and import the new 
> site to there (which has exactly the same content as the existing site).
> 
> 2. I will ask infra to use the new repository as the source of 
> hadoop.apache.org
> 
> 3. I will sync manually all of the changes in the next two months back to the 
> svn site from the git (release announcements, new committers)
> 
> IN CASE OF ANY PROBLEM we can switch back to the svn without any problem.
> 
> If no-one objects within three days, I'll assume lazy consensus and start 
> with this plan. Please comment if you have objections.
> 
> Again: it allows immediate fallback at any time as svn repo will be kept as 
> is (+ I will keep it up-to-date in the next 2 months)
> 
> Thanks,
> Marton
> 
> 
> On 06/21/2018 09:00 PM, Elek, Marton wrote:
>> Thank you very much to bump up this thread.
>> About [2]: (Just for the clarification) the content of the proposed website 
>> is exactly the same as the old one.
>> About [1]. I believe that the "mvn site" is perfect for the documentation 
>> but for website creation there are more simple and powerful tools.
>> Hugo has more simple compared to jekyll. Just one binary, without 
>> dependencies, works everywhere (mac, linux, windows)
>> Hugo has much more powerful compared to "mvn site". Easier to create/use 
>> more modern layout/theme, and easier to handle the content (for example new 
>> release announcements could be generated as part of the release process)
>> I think it's very low risk to try out a new approach for the site (and easy 
>> to rollback in case of problems)
>> Marton
>> ps: I just updated the patch/preview site with the recent releases:
>> ***
>> * http://hadoop.anzix.net *
>> ***
>> On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
>>> Got pinged about this offline.
>>> 
>>> Thanks for keeping at it, Marton!
>>> 
>>> I think there are two road-blocks here
>>>   (1) Is the mechanism using which the website is built good enough - 
>>> mvn-site / hugo etc?
>>>   (2) Is the new website good enough?
>>> 
>>> For (1), I just think we need more committer attention and get feedback 
>>> rapidly and get it in.
>>> 
>>> For (2), how about we do it in a different way in the interest of progress?
>>>   - We create a hadoop.apache.org/new-site/ where this new site goes.
>>>   - We then modify the existing web-site to say that there is a new 
>>> site/experience that folks can click on a link and navigate to
>>>   - As this new website matures and gets feedback & fixes, we finally pull 
>>> the plug at a later point of time when we think we are good to go.
>>> 
>>> Thoughts?
>>> 
>>> +Vinod
>>> 
>>>> On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I would like to bump this thread up.
>>>> 
>>>> TLDR; There is a proposed version of a new hadoop site which is available 
>>>> from here: https://elek.github.io/hadoop-site-proposal/ and 
>>>> https://issues.apache.org/jira/browse/HADOOP-14163
>>>> 
>>>> Please let me know what you think about it.
>>>> 
>>>> 
>>>> Longer version:
>>>> 
>>>> This thread started long time ago to use a more modern hadoop site:
>>>> 
>>>> Goals were:
>>>> 
>>>> 1. To make it easier to manage it (the release entries could be created by 
>>>> a script as part of the release process)
>>>> 2. To use a better look-and-feel
>>>> 3. Move it out from svn to git
>>>> 
>>>> I proposed to:
>>>> 
>>>> 1. Move the existing site to git and generate it with hugo (which is a 
>>>> single, standalone binary)
>>>> 2. Move both the rendered and source branches to git.
>>>> 3. (Create a jenkins job to generate the site automatically)
>>>> 
>>>> NOTE: this is just about forrest based hadoop.apache.org, NOT about the 
>>>> documentation which is generated by mvn-site (as before)
>>>> 
>>>> 
>>>> I got multiple valuable feedback and I improved 

Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)

[Vinod Kumar Vavilapalli]

Yongjun,

Looks like you didn't add the links to 3.0.3 binary release on the 
http://hadoop.apache.org/releases.html page.

I just did it, FYI: 
https://svn.apache.org/viewvc?view=revision=1837967 


Thanks
+Vinod


> On May 31, 2018, at 10:48 PM, Yongjun Zhang  wrote:
> 
> Greetings all,
> 
> I've created the first release candidate (RC0) for Apache Hadoop
> 3.0.3. This is our next maintenance release to follow up 3.0.2. It includes
> about 249
> important fixes and improvements, among which there are 8 blockers. See
> https://issues.apache.org/jira/issues/?filter=12343997
> 
> The RC artifacts are available at:
> https://dist.apache.org/repos/dist/dev/hadoop/3.0.3-RC0/
> 
> The maven artifacts are available via
> https://repository.apache.org/content/repositories/orgapachehadoop-1126
> 
> Please try the release and vote; the vote will run for the usual 5 working
> days, ending on 06/07/2018 PST time. Would really appreciate your
> participation here.
> 
> I bumped into quite some issues along the way, many thanks to quite a few
> people who helped, especially Sammi Chen, Andrew Wang, Junping Du, Eddy Xu.
> 
> Thanks,
> 
> --Yongjun

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Vinod Kumar Vavilapalli]

+1

Thanks
+Vinod

> On Jul 6, 2018, at 11:12 AM, Sunil G  wrote:
> 
> I just checked.  YARN-7556 and YARN-7451 can be cherry-picked.
> I cherry-picked in my local and compiled. Things are good.
> 
> I can push this now  which will restore trunk to its original.
> I can do this if there are no objection.
> 
> - Sunil
> 
> On Fri, Jul 6, 2018 at 11:10 AM Arpit Agarwal  <mailto:aagar...@hortonworks.com>>
> wrote:
> 
>> afaict YARN-8435 is still in trunk. YARN-7556 and YARN-7451 are not.
>> 
>> 
>> From: Giovanni Matteo Fumarola 
>> Date: Friday, July 6, 2018 at 10:59 AM
>> To: Vinod Kumar Vavilapalli 
>> Cc: Anu Engineer , Arpit Agarwal <
>> aagar...@hortonworks.com>, "su...@apache.org" , "
>> yarn-dev@hadoop.apache.org" , "
>> hdfs-...@hadoop.apache.org" , "
>> common-...@hadoop.apache.org" , "
>> mapreduce-...@hadoop.apache.org" 
>> Subject: Re: [VOTE] reset/force push to clean up inadvertent merge commit
>> pushed to trunk
>> 
>> Everything seems ok except the 3 commits: YARN-8435, YARN-7556, YARN-7451
>> are not anymore in trunk due to the revert.
>> 
>> Haibo/Robert if you can recommit your patches I will commit mine
>> subsequently to preserve the original order.
>> 
>> (My apology for the mess I did with the merge commit)
>> 
>> On Fri, Jul 6, 2018 at 10:42 AM, Vinod Kumar Vavilapalli <
>> vino...@apache.org <mailto:vino...@apache.org><mailto:vino...@apache.org 
>> <mailto:vino...@apache.org>>> wrote:
>> I will add that the branch also successfully compiles.
>> 
>> Let's just move forward as is, unblock commits and just fix things if
>> anything is broken.
>> 
>> +Vinod
>> 
>>> On Jul 6, 2018, at 10:30 AM, Anu Engineer >> <mailto:aengin...@hortonworks.com>
>> <mailto:aengin...@hortonworks.com <mailto:aengin...@hortonworks.com>>> wrote:
>>> 
>>> Hi All,
>>> 
>>> [ Thanks to Arpit for working offline and verifying that branch is
>> indeed good.]
>>> 
>>> I want to summarize what I know of this issue and also solicit other
>> points of view.
>>> 
>>> We reverted the commit(c163d1797) from the branch, as soon as we noticed
>> it. That is, we have made no other commits after the merge commit.
>>> 
>>> We used the following command to revert
>>> git revert -c c163d1797ade0f47d35b4a44381b8ef1dfec5b60 -m 1
>>> 
>>> Giovanni's branch had three commits + merge, The JIRAs he had were
>> YARN-7451, YARN-7556, YARN-8435.
>>> 
>>> The issue seems to be the revert of merge has some diffs. I am not a
>> YARN developer, so the only problem is to look at the revert and see if
>> there were any spurious edits in Giovanni's original commit + merge.
>>> If there are none, we don't need a reset/force push.  But if we find an
>> issue I am more than willing to go the force commit route.
>>> 
>>> The revert takes the trunk back to the point of the first commit from
>> Giovanni which is YARN-8435. His branch was also rewriting the order of
>> commits which we have lost due to the revert.
>>> 
>>> Based on what I know so far, I am -1 on the force push.
>>> 
>>> In other words, I am trying to understand why we need the force push. I
>> have left a similar comment in JIRA (
>> https://issues.apache.org/jira/browse/INFRA-16727 
>> <https://issues.apache.org/jira/browse/INFRA-16727>) too.
>>> 
>>> 
>>> Thanks
>>> Anu
>>> 
>>> 
>>> On 7/6/18, 10:24 AM, "Arpit Agarwal" >> <mailto:aagar...@hortonworks.com>> aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>>> wrote:
>>> 
>>>   -1 for the force push. Nothing is broken in trunk. The history looks
>> ugly for two commits and we can live with it.
>>> 
>>>   The revert restored the branch to Giovanni's intent. i.e. only
>> YARN-8435 is applied. Verified there is no delta between hashes 0d9804d and
>> 39ad989 (HEAD).
>>> 
>>>   39ad989 2018-07-05 aengineer@ o {apache/trunk} Revert "Merge branch
>> 't...
>>>   c163d17 2018-07-05 gifuma@apa M─┐ Merge branch 'trunk' of
>> https://git- <https://git-/>...
>>>   99febe7 2018-07-05 rkanter@ap │ o YARN-7451. Add missing tests to
>> veri...
>>>   1726247 2018-07-05 haibochen@ │ o YARN-7556. Fair scheduler
>> configurat...
>>>   0d9804d 2018-07-05 gifuma@apa o │

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Vinod Kumar Vavilapalli]

I will add that the branch also successfully compiles.

Let's just move forward as is, unblock commits and just fix things if anything 
is broken.

+Vinod

> On Jul 6, 2018, at 10:30 AM, Anu Engineer  wrote:
> 
> Hi All,
> 
> [ Thanks to Arpit for working offline and verifying that branch is indeed 
> good.]
> 
> I want to summarize what I know of this issue and also solicit other points 
> of view.
> 
> We reverted the commit(c163d1797) from the branch, as soon as we noticed it. 
> That is, we have made no other commits after the merge commit.
> 
> We used the following command to revert 
> git revert -c c163d1797ade0f47d35b4a44381b8ef1dfec5b60 -m 1
> 
> Giovanni's branch had three commits + merge, The JIRAs he had were YARN-7451, 
> YARN-7556, YARN-8435.
> 
> The issue seems to be the revert of merge has some diffs. I am not a YARN 
> developer, so the only problem is to look at the revert and see if there were 
> any spurious edits in Giovanni's original commit + merge. 
> If there are none, we don't need a reset/force push.  But if we find an issue 
> I am more than willing to go the force commit route.
> 
> The revert takes the trunk back to the point of the first commit from 
> Giovanni which is YARN-8435. His branch was also rewriting the order of 
> commits which we have lost due to the revert.
> 
> Based on what I know so far, I am -1 on the force push.
> 
> In other words, I am trying to understand why we need the force push. I have 
> left a similar comment in JIRA 
> (https://issues.apache.org/jira/browse/INFRA-16727) too.
> 
> 
> Thanks
> Anu
> 
> 
> On 7/6/18, 10:24 AM, "Arpit Agarwal"  wrote:
> 
>-1 for the force push. Nothing is broken in trunk. The history looks ugly 
> for two commits and we can live with it.
> 
>The revert restored the branch to Giovanni's intent. i.e. only YARN-8435 
> is applied. Verified there is no delta between hashes 0d9804d and 39ad989 
> (HEAD).
> 
>39ad989 2018-07-05 aengineer@ o {apache/trunk} Revert "Merge branch 't...
>c163d17 2018-07-05 gifuma@apa M─┐ Merge branch 'trunk' of https://git-...
>99febe7 2018-07-05 rkanter@ap │ o YARN-7451. Add missing tests to veri...
>1726247 2018-07-05 haibochen@ │ o YARN-7556. Fair scheduler configurat...
>0d9804d 2018-07-05 gifuma@apa o │ YARN-8435. Fix NPE when the same cli...
>71df8c2 2018-07-05 nanda@apac o─┘ HDDS-212. Introduce NodeStateManager...
> 
>Regards,
>Arpit
> 
> 
>On 7/5/18, 2:37 PM, "Subru Krishnan"  wrote:
> 
>Folks,
> 
>There was a merge commit accidentally pushed to trunk, you can find the
>details in the mail thread [1].
> 
>I have raised an INFRA ticket [2] to reset/force push to clean up 
> trunk.
> 
>Can we have a quick vote for INFRA sign-off to proceed as this is 
> blocking
>all commits?
> 
>Thanks,
>Subru
> 
>[1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201807.mbox/%3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%40mail.gmail.com%3E
>[2] https://issues.apache.org/jira/browse/INFRA-16727
> 
> 
> 
>-
>To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 
> 
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Merge branch commit in trunk by mistake

[Vinod Kumar Vavilapalli]

What is broken due to this merge commit?

+Vinod

> On Jul 5, 2018, at 2:03 PM, Arun Suresh  wrote:
> 
> I agree with Sean, to be honest.. it is disruptive.
> Also, we have to kind of lock down the repo till it is completed..
> 
> I recommend we be careful and try not to get into this situation again..
> 
> -1 on force pushing..
> 
> Cheers
> -Arun
> 
> On Thu, Jul 5, 2018, 1:55 PM Sean Busbey  wrote:
> 
>> If we need a vote, please have a thread with either DISCUSS or
>> preferably VOTE in the subject so folks are more likely to see it.
>> 
>> that said, I'm -1 (non-binding). force pushes are extremely
>> disruptive. there's no way to know who's updated their local git repo
>> to include these changes in the last few hours. if a merge commit is
>> so disruptive that we need to subject folks to the inconvenience of a
>> force push then we should have more tooling in place to avoid them
>> (like client side git hooks for all committers).
>> 
>> On Thu, Jul 5, 2018 at 3:36 PM, Wangda Tan  wrote:
>>> +1 for force reset the branch.
>>> 
>>> On Thu, Jul 5, 2018 at 12:14 PM Subru Krishnan  wrote:
>>> 
 Looking at the merge commit, I feel it's better to reset/force push
 especially since this is still the latest commit on trunk.
 
 I have raised an INFRA ticket requesting the same:
 https://issues.apache.org/jira/browse/INFRA-16727
 
 -S
 
 On Thu, Jul 5, 2018 at 11:45 AM, Sean Busbey
>> 
 wrote:
 
> FYI, no images make it through ASF mailing lists. I presume the image
>> was
> of the git history? If that's correct, here's what that looks like in
>> a
> paste:
> 
> https://paste.apache.org/eRix
> 
> There are no force pushes on trunk, so backing the change out would
 require
> the PMC asking INFRA to unblock force pushes for a period of time.
> 
> Probably the merge commit isn't a big enough deal to do that. There
>> was a
> merge commit ~5 months ago for when YARN-6592 merged into trunk.
> 
> So I'd say just try to avoid doing it in the future?
> 
> -busbey
> 
> On Thu, Jul 5, 2018 at 1:31 PM, Giovanni Matteo Fumarola <
> giovanni.fumar...@gmail.com> wrote:
> 
>> Hi folks,
>> 
>> After I pushed something on trunk a merge commit showed up in the
> history. *My
>> bad*.
>> 
>> 
>> 
>> Since it was one of my first patches, I run a few tests on my
>> machine
>> before checked in.
>> While I was running all the tests, someone else checked in. I
>> correctly
>> pulled all the new changes.
>> 
>> Even before I did the "git push" there was no merge commit in my
 history.
>> 
>> Can someone help me reverting this change?
>> 
>> Thanks
>> Giovanni
>> 
>> 
>> 
> 
> 
> --
> busbey
> 
 
>> 
>> 
>> 
>> --
>> busbey
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8338) TimelineService V1.5 doesn't come up after HADOOP-15406

[Vinod Kumar Vavilapalli (JIRA)]

Vinod Kumar Vavilapalli created YARN-8338:
-

 Summary: TimelineService V1.5 doesn't come up after HADOOP-15406
 Key: YARN-8338
 URL: https://issues.apache.org/jira/browse/YARN-8338
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli


TimelineService V1.5 fails with the following:

{code}
java.lang.NoClassDefFoundError: org/objenesis/Objenesis
at 
org.apache.hadoop.yarn.server.timeline.RollingLevelDBTimelineStore.(RollingLevelDBTimelineStore.java:174)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-8319) More YARN pages need to honor yarn.resourcemanager.display.per-user-apps

[Vinod Kumar Vavilapalli (JIRA)]

Vinod Kumar Vavilapalli created YARN-8319:
-

 Summary: More YARN pages need to honor 
yarn.resourcemanager.display.per-user-apps
 Key: YARN-8319
 URL: https://issues.apache.org/jira/browse/YARN-8319
 Project: Hadoop YARN
  Issue Type: Bug
  Components: webapp
Reporter: Vinod Kumar Vavilapalli


When this config is on
 - Per queue page on UI2 should filter app list by user
 -- TODO: Verify the same with UI1 Per-queue page
 - ATSv2 with UI2 should filter list of all users' flows and flow activities
 - Per Node pages
 -- Listing of apps and containers on a per-node basis should filter apps and 
containers by user.

To this end, because this is no longer just for resourcemanager, we should also 
deprecate {{yarn.resourcemanager.display.per-user-apps}} in favor of 
{{yarn.webapp.filter-app-list-by-user}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-8181) Docker container run_time

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-8181.
---
Resolution: Invalid

[~sajavadi], please see http://hadoop.apache.org/mailing_lists.html. You can 
send emails to u...@hadoop.apache.org. You can subscribe to the list for other 
related discussions.

Resolving this for now.

> Docker container run_time
> -
>
> Key: YARN-8181
> URL: https://issues.apache.org/jira/browse/YARN-8181
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Seyyed Ahmad Javadi
>Priority: Major
>
> Hi All,
> I want to use docker container run time but could not solve the facing 
> problem. I am following the guide below and the NM log is as follows. I can 
> not see any docker containers to be created. It works when I use default LCE. 
> Please also find how I submit a job at the end as well.
> Do you have any guide on how can I make Docker rum_time works?
> May you please let me know how can use LCE binary to make sure my docker 
> setup is correct?
> I confirmed that "docker run" works fine. I really like this developing 
> feature and would like to contribute to it. Many thanks in advance.
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/DockerContainers.html]
> {code:java}
> NM LOG:
> ...
> 2018-04-19 11:49:24,568 INFO SecurityLogger.org.apache.hadoop.ipc.Server: 
> Auth successful for appattempt_1524151293356_0005_01 (auth:SIMPLE)
> 2018-04-19 11:49:24,580 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>  Start request for container_1524151293356_0005_01_01 by user ubuntu
> 2018-04-19 11:49:24,584 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>  Creating a new application reference for app application_1524151293356_0005
> 2018-04-19 11:49:24,584 INFO 
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ubuntu    
> IP=130.245.127.176    OPERATION=Start Container Request    
> TARGET=ContainerManageImpl    RESULT=SUCCESS    
> APPID=application_1524151293356_0005    
> CONTAINERID=container_1524151293356_0005_01_01
> 2018-04-19 11:49:24,585 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
>  Application application_1524151293356_0005 transitioned from NEW to INITING
> 2018-04-19 11:49:24,585 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
>  Adding container_1524151293356_0005_01_01 to application 
> application_1524151293356_0005
> 2018-04-19 11:49:24,585 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
>  Application application_1524151293356_0005 transitioned from INITING to 
> RUNNING
> 2018-04-19 11:49:24,588 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
>  Container container_1524151293356_0005_01_01 transitioned from NEW to 
> LOCALIZING
> 2018-04-19 11:49:24,588 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got 
> event CONTAINER_INIT for appId application_1524151293356_0005
> 2018-04-19 11:49:24,589 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>  Created localizer for container_1524151293356_0005_01_01
> 2018-04-19 11:49:24,616 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
>  Writing credentials to the nmPrivate file 
> /tmp/hadoop-ubuntu/nm-local-dir/nmPrivate/container_1524151293356_0005_01_01.tokens
> 2018-04-19 11:49:28,090 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
>  Container container_1524151293356_0005_01_01 transitioned from 
> LOCALIZING to SCHEDULED
> 2018-04-19 11:49:28,090 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerScheduler:
>  Starting container [container_1524151293356_0005_01_01]
> 2018-04-19 11:49:28,212 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
>  Container container_1524151293356_0005_01_01 transitioned from SCHEDULED 
> to RUNNING
> 2018-04-19 11:49:28,212 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  Starting resource-monitoring for container_1524151293356_0005_01_01
> 2018-04-19 11:49:29,401 INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
>  Cont

[jira] [Resolved] (YARN-7212) [Atsv2] TimelineSchemaCreator fails to create flowrun table causes RegionServer down!

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-7212.
---
Resolution: Duplicate

> [Atsv2] TimelineSchemaCreator fails to create flowrun table causes 
> RegionServer down!
> -
>
> Key: YARN-7212
> URL: https://issues.apache.org/jira/browse/YARN-7212
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Priority: Major
>
> *Hbase-2.0* officially support *hadoop-alpha* compilations. So I was trying 
> to build and test with HBase-2.0. But table schema creation fails and causes 
> RegionServer to shutdown with following error
> {noformat}
> Caused by: java.lang.NoSuchMethodError: 
> org.apache.hadoop.hbase.Tag.asList([BII)Ljava/util/List;
> at 
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowScanner.getCurrentAggOp(FlowScanner.java:250)
> at 
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowScanner.nextInternal(FlowScanner.java:226)
> at 
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowScanner.next(FlowScanner.java:145)
> at 
> org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:132)
> at 
> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:973)
> at 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2252)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2672)
> {noformat}
> Since HBase-2.0 community is ready to release Hadoop-3.x compatible versions, 
> ATSv2 also need to support HBase-2.0 versions. For this, we need to take up a 
> task of test and validate HBase-2.0 issues! 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-1489) [Umbrella] Work-preserving ApplicationMaster restart

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-1489.
---
Resolution: Fixed
  Assignee: (was: Vinod Kumar Vavilapalli)

Resolved this very old feature as fixed. Keeping it unassigned given multiple 
contributors. No fix-version given the tasks (perhaps?) spanned across releases.

> [Umbrella] Work-preserving ApplicationMaster restart
> 
>
> Key: YARN-1489
> URL: https://issues.apache.org/jira/browse/YARN-1489
> Project: Hadoop YARN
>  Issue Type: Bug
>        Reporter: Vinod Kumar Vavilapalli
>Priority: Major
> Attachments: Work preserving AM restart.pdf
>
>
> Today if AMs go down,
>  - RM kills all the containers of that ApplicationAttempt
>  - New ApplicationAttempt doesn't know where the previous containers are 
> running
>  - Old running containers don't know where the new AM is running.
> We need to fix this to enable work-preserving AM restart. The later two 
> potentially can be done at the app level, but it is good to have a common 
> solution for all apps where-ever possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-5881.
---
Resolution: Fixed

> Enable configuration of queue capacity in terms of absolute resources
> -
>
> Key: YARN-5881
> URL: https://issues.apache.org/jira/browse/YARN-5881
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Sean Po
>Assignee: Sunil G
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: 
> YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf,
>  YARN-5881.v0.patch, YARN-5881.v1.patch
>
>
> Currently, Yarn RM supports the configuration of queue capacity in terms of a 
> proportion to cluster capacity. In the context of Yarn being used as a public 
> cloud service, it makes more sense if queues can be configured absolutely. 
> This will allow administrators to set usage limits more concretely and 
> simplify customer expectations for cluster allocation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-6223) [Umbrella] Natively support GPU configuration/discovery/scheduling/isolation on YARN

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-6223.
---
Resolution: Fixed

> [Umbrella] Natively support GPU configuration/discovery/scheduling/isolation 
> on YARN
> 
>
> Key: YARN-6223
> URL: https://issues.apache.org/jira/browse/YARN-6223
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-6223.Natively-support-GPU-on-YARN-v1.pdf, 
> YARN-6223.wip.1.patch, YARN-6223.wip.2.patch, YARN-6223.wip.3.patch
>
>
> As varieties of workloads are moving to YARN, including machine learning / 
> deep learning which can speed up by leveraging GPU computation power. 
> Workloads should be able to request GPU from YARN as simple as CPU and memory.
> *To make a complete GPU story, we should support following pieces:*
> 1) GPU discovery/configuration: Admin can either config GPU resources and 
> architectures on each node, or more advanced, NodeManager can automatically 
> discover GPU resources and architectures and report to ResourceManager 
> 2) GPU scheduling: YARN scheduler should account GPU as a resource type just 
> like CPU and memory.
> 3) GPU isolation/monitoring: once launch a task with GPU resources, 
> NodeManager should properly isolate and monitor task's resource usage.
> For #2, YARN-3926 can support it natively. For #3, YARN-3611 has introduced 
> an extensible framework to support isolation for different resource types and 
> different runtimes.
> *Related JIRAs:*
> There're a couple of JIRAs (YARN-4122/YARN-5517) filed with similar goals but 
> different solutions:
> For scheduling:
> - YARN-4122/YARN-5517 are all adding a new GPU resource type to Resource 
> protocol instead of leveraging YARN-3926.
> For isolation:
> - And YARN-4122 proposed to use CGroups to do isolation which cannot solve 
> the problem listed at 
> https://github.com/NVIDIA/nvidia-docker/wiki/GPU-isolation#challenges such as 
> minor device number mapping; load nvidia_uvm module; mismatch of CUDA/driver 
> versions, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-5983) [Umbrella] Support for FPGA as a Resource in YARN

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-5983.
---
Resolution: Fixed

> [Umbrella] Support for FPGA as a Resource in YARN
> -
>
> Key: YARN-5983
> URL: https://issues.apache.org/jira/browse/YARN-5983
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: yarn
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-5983-Support-FPGA-resource-on-NM-side_v1.pdf, 
> YARN-5983-implementation-notes.pdf, YARN-5983_end-to-end_test_report.pdf
>
>
> As various big data workload running on YARN, CPU will no longer scale 
> eventually and heterogeneous systems will become more important. ML/DL is a 
> rising star in recent years, applications focused on these areas have to 
> utilize GPU or FPGA to boost performance. Also, hardware vendors such as 
> Intel also invest in such hardware. It is most likely that FPGA will become 
> popular in data centers like CPU in the near future.
> So YARN as a resource managing and scheduling system, would be great to 
> evolve to support this. This JIRA proposes FPGA to be a first-class citizen. 
> The changes roughly includes:
> 1. FPGA resource detection and heartbeat
> 2. Scheduler changes (YARN-3926 invlolved)
> 3. FPGA related preparation and isolation before launch container
> We know that YARN-3926 is trying to extend current resource model. But still 
> we can leave some FPGA related discussion here



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Vinod Kumar Vavilapalli]

That is a great observation. And I missed your previous email about the shaded 
vs unshaded jars already getting fixed.

I guess we are good to go.

--

Looking at the RC. Went through my usual check-list. Here's my summary.

Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
-- Note: The format of the mds files changed a bit - not a biggie.
-- For e.g, in 3.0.0 and 2.x releases, it has lines of the form 
"hadoop-3.0.0-src.tar.gz: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279 7BCED50B 
B2600318 B7E0BE3A 74DFFF71"
-- But in 3.1.0 RC it is, 
"/build/source/target/artifacts/hadoop-3.1.0.tar.gz: SHA256 = 670D2CED 595FA42D 
9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
- [Check] Generating dist tarballs from source tar-ball
- [Check] Testing
   -- Start NN, DN, RM, NM, JHS, Timeline Service
   -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
   -- Tested CLIs to print nodes, apps etc and also navigated UIs

+1 binding.

Thanks
+Vinod

> On Apr 3, 2018, at 8:13 PM, Wangda Tan <wheele...@gmail.com> wrote:
> 
> Hi Vinod / Arpit,
> 
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
> 
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
> 
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
> 
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
> 
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
> 
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 289
> 
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
> 
> I guess the differences come from our release process.
> 
> Thanks,
> Wangda
> 
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/
> /hadoop-yarn-api-.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
> 
> 
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vino...@apache.org>
> wrote:
> 
>> We vote on the source code. The binaries are convenience artifacts.
>> 
>> This is what I would do - (a) Just replace both the maven jars as well as
>> the binaries to be consistent and correct. And then (b) Give a couple more
>> days for folks who tested on the binaries to reverify - I count one such
>> clear vote as of now.
>> 
>> Thanks
>> +Vinod
>> 
>> 
>> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
>> 
>> HI Arpit,
>> 
>> I think it won't match if we do rebuild. It should be fine as far as
>> they're signed, correct? I don't see any policy doesn't allow this.
>> 
>> Thanks,
>> Wangda
>> 
>> 
>> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com>
>> wrote:
>> 
>>> Thanks Wangda, I see the shaded jars now.
>>> 
>>> Are the repo jars required to be the same as the binary release? They
>>> don’t match right now, probably they got rebuilt.
>>> 
>>> +1 (binding), modulo that remaining question.
>>> 
>>> * Verified signatures
>>> * Verified checksums for source and binary artefacts
>>> * Sanity checked jars on r.a.o.
>>> * Built from source
>>> * Deployed to 3 node secure cluster with NameNode HA
>>> * Verified HDFS web UIs
>>> * Tried out HDFS shell commands
>>> * Ran sample MapReduce jobs
>>> 
>>> Thanks!
>>> 
>>> 
>>> ----------
>>> From: Wangda Tan <wheele...@gmail.com>
>>> Date: Monday, April 2, 2018 at 9:25 PM
>>> To: Arpit Agarwal <aagar...@hortonworks.com>
>>> Cc: Gera Shegalov <ger...@gmail.com>, Sunil G <sun...@apache.org>, "
>>> yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>, Hdfs-dev <
>>> hdfs-...@hadoop.apache.org>, Hadoop Common <common-...@hadoop.apache.org>,
>>> "mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>,
>>> Vinod Kumar Vavilapalli <vino...@apache.org>
>>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>>> 
>>> As point

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Vinod Kumar Vavilapalli]

We vote on the source code. The binaries are convenience artifacts.

This is what I would do - (a) Just replace both the maven jars as well as the 
binaries to be consistent and correct. And then (b) Give a couple more days for 
folks who tested on the binaries to reverify - I count one such clear vote as 
of now.

Thanks
+Vinod

> On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
> 
> HI Arpit,
> 
> I think it won't match if we do rebuild. It should be fine as far as they're 
> signed, correct? I don't see any policy doesn't allow this. 
> 
> Thanks,
> Wangda
> 
> 
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com 
> <mailto:aagar...@hortonworks.com>> wrote:
> Thanks Wangda, I see the shaded jars now.
> 
> Are the repo jars required to be the same as the binary release? They don’t 
> match right now, probably they got rebuilt.
> 
> +1 (binding), modulo that remaining question.
> 
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
> 
> Thanks!
> 
> 
> --
> From: Wangda Tan <wheele...@gmail.com <mailto:wheele...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal <aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>>
> Cc: Gera Shegalov <ger...@gmail.com <mailto:ger...@gmail.com>>, Sunil G 
> <sun...@apache.org <mailto:sun...@apache.org>>, "yarn-dev@hadoop.apache.org 
> <mailto:yarn-dev@hadoop.apache.org>" <yarn-dev@hadoop.apache.org 
> <mailto:yarn-dev@hadoop.apache.org>>, Hdfs-dev <hdfs-...@hadoop.apache.org 
> <mailto:hdfs-...@hadoop.apache.org>>, Hadoop Common 
> <common-...@hadoop.apache.org <mailto:common-...@hadoop.apache.org>>, 
> "mapreduce-...@hadoop.apache.org <mailto:mapreduce-...@hadoop.apache.org>" 
> <mapreduce-...@hadoop.apache.org <mailto:mapreduce-...@hadoop.apache.org>>, 
> Vinod Kumar Vavilapalli <vino...@apache.org <mailto:vino...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
> 
> As pointed by Arpit, the previously deployed shared jars are incorrect. Just 
> redeployed jars and staged. @Arpit, could you please check the updated Maven 
> repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1092>
> 
> Since the jars inside binary tarballs are correct 
> (http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ 
> <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>). I think we don't need 
> roll another RC, just update Maven repo should be sufficient. 
> 
> Best,
> Wangda
> 
> 
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com 
> <mailto:wheele...@gmail.com>> wrote:
> Hi Arpit,
> 
> Thanks for pointing out this.
> 
> I just removed all .md5 files from artifacts. I found md5 checksums still 
> exist in .mds files and I didn't remove them from .mds file because it is 
> generated by create-release script and Apache guidance is "should not" 
> instead of "must not". Please let me know if you think they need to be 
> removed as well. 
> 
> - Wangda
> 
> 
> 
> On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal 
> <mailto:aagar...@hortonworks.com <mailto:aagar...@hortonworks.com>> wrote:
> Thanks for putting together this RC, Wangda.
> 
> The guidance from Apache is to omit MD5s, specifically:
>   > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).
> 
> https://www.apache.org/dev/release-distribution#sigs-and-sums 
> <https://www.apache.org/dev/release-distribution#sigs-and-sums>
> 
>  
> 
> 
> On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com 
> <mailto:wheele...@gmail.com>> wrote:
> 
> Hi Gera,
> 
> It's my bad, I thought only src/bin tarball is enough.
> 
> I just uploaded all other things under artifact/ to
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ 
> <http://people.apache.org/~wangda/hadoop-3.1.0-RC1/>
> 
> Please let me know if you have any other comments.
> 
> Thanks,
> Wangda
> 
> 
> On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com 
> <mailto:ger...@gmail.com>> wrote:
> 
> 
> Thanks, Wangda!
> 
> There are many more artifacts in previou

[jira] [Resolved] (YARN-8109) Resource Manager WebApps fails to start due to ConcurrentModificationException

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-8109.
---
Resolution: Duplicate

Already fixed by HADOOP-13556. Closing as a dup. Please reopen if that isn't 
the case.

> Resource Manager WebApps  fails to start due to 
> ConcurrentModificationException
> ---
>
> Key: YARN-8109
> URL: https://issues.apache.org/jira/browse/YARN-8109
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Wangda Tan
>Priority: Major
>
> {code}
> 2018-03-22 04:57:39,289 INFO  resourcemanager.ResourceTrackerService 
> (ResourceTrackerService.java:nodeHeartbeat(497)) - Node not found resyncing 
> ctr-e138-1518143905142-129550-01-36.hwx.site:25454
> 2018-03-22 04:57:39,294 INFO  service.AbstractService 
> (AbstractService.java:noteFailure(272)) - Service ResourceManager failed in 
> state STARTED; cause: java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at java.util.Hashtable$Enumerator.next(Hashtable.java:1378)
> at 
> org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2564)
> at 
> org.apache.hadoop.conf.Configuration.getPropsWithPrefix(Configuration.java:2583)
> at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.getConfigParameters(WebApps.java:386)
> at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:334)
> at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:395)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1049)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1152)
> at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1293)
> 2018-03-22 04:57:39,296 INFO  ipc.Server (Server.java:stop(2752)) - Stopping 
> server on 8050
> 2018-03-22 04:57:39,300 INFO  ipc.Server (Server.java:run(932)) - Stopping 
> IPC Server listener on 8050
> 2018-03-22 04:57:39,301 INFO  ipc.Server (Server.java:run(1069)) - Stopping 
> IPC Server Responder
> {code} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-7873) Revert YARN-6078

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-7873.
---
   Resolution: Fixed
Fix Version/s: 3.0.1
   2.9.1
   2.10.0
   3.1.0

> Revert YARN-6078
> 
>
> Key: YARN-7873
> URL: https://issues.apache.org/jira/browse/YARN-7873
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Blocker
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
>
> I think we should revert YARN-6078, since it is not working as intended. The 
> NM does not have permission to destroy the process of the ContainerLocalizer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-7064) Use cgroup to get container resource utilization

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-7064.
---
Resolution: Fixed

Resolving as Fixed instead of Done per our conventions.

> Use cgroup to get container resource utilization
> 
>
> Key: YARN-7064
> URL: https://issues.apache.org/jira/browse/YARN-7064
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7064.000.patch, YARN-7064.001.patch, 
> YARN-7064.002.patch, YARN-7064.003.patch, YARN-7064.004.patch, 
> YARN-7064.005.patch, YARN-7064.007.patch, YARN-7064.008.patch, 
> YARN-7064.009.patch, YARN-7064.010.patch, YARN-7064.011.patch, 
> YARN-7064.012.patch, YARN-7064.013.patch, YARN-7064.014.patch
>
>
> This is an addendum to YARN-6668. What happens is that that jira always wants 
> to rebase patches against YARN-1011 instead of trunk.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About reset branch-3.1 to trunk before release.

[Vinod Kumar Vavilapalli]

Thanks for reporting this.

It's straight forward. Done, commit pushed.

Thanks
+Vinod

> On Mar 19, 2018, at 2:54 PM, Jonathan Kelly <jonathaka...@gmail.com> wrote:
> 
> I just pulled the latest from branch-3.1 and noticed that now that it was
> reset to trunk, the version in the pom.xml files is 3.2.0-SNAPSHOT instead
> of 3.1.0-SNAPSHOT. Is somebody going to submit a new commit to change this
> version back to 3.1.0-SNAPSHOT in this branch?
> 
> Thank you,
> Jonathan
> 
> On Mon, Mar 19, 2018 at 11:43 AM Arpit Agarwal <aagar...@hortonworks.com>
> wrote:
> 
>> Thanks Wangda.
>> 
>> 
>> On 3/19/18, 11:38 AM, "Wangda Tan" <wheele...@gmail.com> wrote:
>> 
>>Done JIRA fix version update:
>> 
>>Moved all JIRAs with fixVersion = 3.2.0 to 3.1.0 except following few
>> fixes
>>(which committed after 49c747ab187d0650143205ba57ca19607ec4c6bd)
>> 
>>YARN-8002. Support NOT_SELF and ALL namespace types for allocation
>> tag.
>>(Weiwe
>>i Yang via wangda)
>> 
>>HADOOP-15262. AliyunOSS: move files under a directory in parallel
>> when
>>rename a directory. Contributed by Jinhu Wu.
>> 
>>MAPREDUCE-7066. TestQueue fails on Java9
>> 
>>YARN-8028. Support authorizeUserAccessToQueue in RMWebServices.
>>Contributed by Wangda Tan.
>> 
>>YARN-8040. [UI2] New YARN UI webapp does not respect current
>> pathname
>>for REST api. Contributed by Sunil G.
>> 
>>Thanks,
>>Wangda
>> 
>>On Mon, Mar 19, 2018 at 11:12 AM, Wangda Tan <wheele...@gmail.com>
>> wrote:
>> 
>>> Thanks Akira for the additional vote,
>>> 
>>> With help from Apache Infra Team (Daniel Takamori), we just reset
>>> branch-3.1 to trunk (SHA: 49c747ab187d0650143205ba57ca19607ec4c6bd).
>> Will
>>> update JIRA fix version shortly.
>>> 
>>> - Wangda
>>> 
>>> On Sun, Mar 18, 2018 at 6:10 PM, Akira Ajisaka <
>> ajisa...@oss.nttdata.co.jp
>>>> wrote:
>>> 
>>>> +1 for resetting branch-3.1.
>>>> 
>>>> Thanks,
>>>> Akira
>>>> 
>>>> 
>>>> On 2018/03/18 12:51, Wangda Tan wrote:
>>>> 
>>>>> Thanks for sharing your thoughts.
>>>>> 
>>>>> We have done build and single node cluster deploy / test for the
>> latest
>>>>> trunk code (commit: 49c747ab187d0650143205ba57ca19607ec4c6bd).
>> Since
>>>>> there
>>>>> are no objections, so I will go ahead to do the branch replace.
>>>>> 
>>>>> Since we don't have force push permission to release branches. I
>> just
>>>>> filed
>>>>> https://issues.apache.org/jira/browse/INFRA-16204 to get help from
>>>>> Apache
>>>>> infra team.
>>>>> 
>>>>> Please hold any commits to branch-3.1, will keep this email thread
>>>>> posted.
>>>>> 
>>>>> Best,
>>>>> Wangda
>>>>> 
>>>>> On Wed, Mar 14, 2018 at 3:14 PM, Vinod Kumar Vavilapalli <
>>>>> vino...@apache.org
>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>> 
>>>>> I see one new feature:
>> https://issues.apache.org/jira/browse/YARN-7626:
>>>>>> Allow regular expression matching in container-executor.cfg for
>> devices
>>>>>> and
>>>>>> named docker volumes mount.
>>>>>> 
>>>>>> There are 21 sub-tasks. There are three feature-type JIRAs in
>> those -
>>>>>> https://issues.apache.org/jira/browse/YARN-7972,
>>>>>> https://issues.apache.org/jira/browse/YARN-7891 and
>>>>>> https://issues.apache.org/jira/browse/YARN-5015. These should be
>> okay -
>>>>>> not major disrupting features.
>>>>>> 
>>>>>> Everything else is either a bug-fix or an improvement so we
>> should be
>>>>>> good.
>>>>>> 
>>>>>> From the list, it doesn't look like resetting will destabilize
>> 3.1, +1
>>>>>> for
>>>>>> doing this.
>>>>>> 
>>>>>> Thanks
>>>>>> +Vinod
>>>>>> 
>>>>>> On Mar

Re: About reset branch-3.1 to trunk before release.

[Vinod Kumar Vavilapalli]

I see one new feature: https://issues.apache.org/jira/browse/YARN-7626: Allow 
regular expression matching in container-executor.cfg for devices and named 
docker volumes mount.

There are 21 sub-tasks. There are three feature-type JIRAs in those - 
https://issues.apache.org/jira/browse/YARN-7972, 
https://issues.apache.org/jira/browse/YARN-7891 and 
https://issues.apache.org/jira/browse/YARN-5015. These should be okay - not 
major disrupting features.

Everything else is either a bug-fix or an improvement so we should be good.

From the list, it doesn't look like resetting will destabilize 3.1, +1 for 
doing this.

Thanks
+Vinod

> On Mar 14, 2018, at 1:54 PM, Wangda Tan  wrote:
> 
> Hi mapreduce/yarn/common/hdfs-devs, 
> 
> As of now, we have all blockers done for 3.1.0 release [1]. The release is 
> running behind schedule due to a few security-related issues. Because of this 
> and since branch-3.1 is cut 5 weeks before on Feb 8, trunk 3.2 is already 
> diverging. There're 64 commits in trunk but not in branch-3.1. [2]
> 
> I took a quick scan of them, most of them are good fixes which we should 
> bring to 3.1.0 as well. And this can also reduce differences between 3.2.0 
> and 3.1.0 release for less maintenance burden in the future.
> 
> Unless anyone objects, we will reset branch-3.1 to trunk in 1-2 days and cut 
> RC after that.
> 
> Thoughts?
> 
> - Wangda
> 
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, 
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.1.0 ORDER BY 
> priority DESC
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0) AND 
> fixVersion not in (3.1.0)


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Vinod Kumar Vavilapalli]

Was out of country and away from email for weeks.

We have been using this in the past: 
https://www.meetup.com/Hadoop-Contributors/ 
. I believe this doesn't have the 
per-meetup-charge issue, but will double check. Let me know if there are more 
meetups planned in the near future and we can use this.

Thanks
+Vinod

> On Mar 6, 2018, at 7:48 PM, Chris Douglas  wrote:
> 
> Found a meetup alternative (thanks Subru):
> https://meetingstar.io/event/fk13172f1d75KN 
> 
> 
> So we can get a rough headcount, please add (local) if you plan to
> attend in-person. -C
> 
> 
> On Mon, Mar 5, 2018 at 4:03 PM, Chris Douglas  > wrote:
>> [Cross-posting, as this affects the rest of the project]
>> 
>> Hey folks-
>> 
>> As discussed last month [1], the HDFS build hasn't been healthy
>> recently. We're dedicating a bug bash to stabilize the build and
>> address some longstanding issues with our unit tests. We rely on our
>> CI infrastructure to keep the project releasable, and in its current
>> state, it's not protecting us from regressions. While we probably
>> won't achieve all our goals in this session, we can develop the
>> conditions for reestablishing a firm foundation.
>> 
>> If you're new to the project, please consider attending and
>> contributing. Committers often prioritize large or complicated
>> patches, and the issues that make the project livable don't get enough
>> attention. A bug bash is a great opportunity to pull reviewers'
>> collars, and fix the annoyances that slow us all down.
>> 
>> If you're a committer, please join us! While some of the proposed
>> repairs are rote, many unit tests rely on implementation details and
>> non-obvious invariants. We need domain experts to help untangle
>> complex dependencies and to prevent breakage of deliberate, but
>> counter-intuitive code.
>> 
>> We're collecting tasks in wiki [2] and will include a dial-in option
>> for folks who aren't local.
>> 
>> Meetup has started charging for creating new events, so we'll have to
>> find another way to get an approximate headcount and publish the
>> address. Please ping me if you have a preferred alternative. -C
>> 
>> [1]: https://s.apache.org/nEoQ
>> [2]: 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org 
> 
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org 
> 

Re: Apache YARN Committers & Contributors Meetup #5

[Vinod Kumar Vavilapalli]

Thanks for that update, Sunil! Good to hear that we had such a successful 
meet-up in Bangalore! Hoping we get back to a regular cadence!

Is it worth doing a simple blog post on this at 
https://blogs.apache.org/hadoop/  , may be 
with pics etc?

Plus Daniel explicitly too. Daniel, can you post the minutes from our YARN 
meetup in the Western hemisphere too? Shall we do a blog post on that too or 
combine it with Sunil's?

Thanks
+Vinod

> On Dec 14, 2017, at 11:08 AM, Sunil G  wrote:
> 
> Hi All,
> 
> Bangalore Hadoop Meetup was recently conducted @Microsoft campus on Dec 8th
> almost in line with YARN Contributors Meetup @CA. Many thanks to Newton
> from MS for organizing this quickly.
> 
> Quick Update about this:
> 1. We had *150*+ participants for this event which was one of the best
> turnout for Hadoop specific meetup in recent times.
> 2. There were 5 talks in total and majority of the talks were on YARN
> (scale/GPU etc). Also shared meetup link below for detailed schedule.
> 3. We were happy to see a excellent turn out from all Hadoop/YARN
> committers and contributors @Bangalore for this event.
> 4. Discussed majorly about upcoming Hadoop 3.0 and next major releases in
> same line. Lot of interests on Resource types / GPU scheduling / EC etc and
> also doubts about rolling upgrade to Hadoop 3.
> 
> Meetup Link: (Slides and Videos will be shared in same link shortly)
> http://meetup.com/Bangalore-Hadoop-Meetups/events/245123037/
> 
> Thanks once again to all folks who joined. We will plan a Bug Bash in same
> timeline when Meetup #5 is going to happen at CA.
> 
> Warm Regards,
> Sunil G
> 
> 
> 
> On Sat, Dec 2, 2017 at 3:55 AM Daniel Templeton  wrote:
> 
>> And thanks to Vinod we now have an official Meetup online:
>> 
>> https://www.meetup.com/Hadoop-Contributors/events/245569075/
>> 
>> Daniel
>> 
>> On 11/22/17 12:20 PM, Daniel Templeton wrote:
>>> We're long past due for another contributors meetup.  Cloudera is
>>> excited to host this event at our new Palo Alto Galactic Headquarters
>>> (
>> https://www.google.com/maps/place/Cloudera+Galactic+HQ/@37.4254615,-122.1413431,17z
>> )
>>> on December 6th from 2pm-5pm PST.  We will have a Google Hangout set
>>> up so that remote participants can dial in.  We will also provide
>>> drinks and snacks.
>>> 
>>> Our agenda will be roughly:
>>> 
>>> * Celebrate Hadoop 3.0.0, identify any loose ends, and talk about 3.0.1
>>> * Planning around Hadoop 3.1.0
>>> * Planning around the branch-2 releases, Hadoop 2.10.0, etc.
>>> * Key signing
>>> * Plan for the next bug bash
>>> 
>>> Hope to see you there!
>>> Daniel
>>> 
>>> PS: I wasn't able to figure out how to create an event in our meetup
>>> group for this.  Anyone want to give me a pointer or set it up?
>> 
>> 
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>> 
>> 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

Yes, JIRAs will be filed, the wiki-page idea from YARN meetup is to record all 
combinations of testing that need to be done and correspondingly capture all 
the testing that someone in the community has already done and record it for 
future perusal.

From what you are saying, I guess we haven't advertised to the public yet on 
rolling upgrades, but in our meetups etc so far, you have been saying that 
rolling upgrades is supported - so I assumed we did put it in our messaging.

The important question is if we are or are not allowed to make potentially 
incompatible changes to fix bugs in the process of supporting 2.x to 3.x 
upgrades whether rolling or not.

+Vinod

> On Dec 13, 2017, at 1:05 PM, Andrew Wang  wrote:
> 
> I'm hoping we can address YARN-7588 and any remaining rolling upgrade issues 
> in 3.0.x maintenance releases. Beyond a wiki page, it would be really great 
> to get JIRAs filed and targeted for tracking as soon as possible.
> 
> Vinod, what do you think we need to do regarding caveating rolling upgrade 
> support? We haven't advertised rolling upgrade support between major releases 
> outside of dev lists and JIRA. As a new major release, our compat guidelines 
> allow us to break compatibility, so I don't think it's expected by users.
> 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

Good stuff Andrew, and thanks everyone!

+Vinod

> On Dec 13, 2017, at 1:05 PM, Andrew Wang  wrote:
> 
> To close this out, the vote passes successfully with 13 binding +1s, 5 
> non-binding +1s, and no -1s. Thanks everyone for voting! I'll work on staging.
> 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

I was waiting for Daniel to post the minutes from YARN meetup to talk about 
this. Anyways, in that discussion, we identified a bunch of key upgrade related 
scenarios that no-one seems to have validated - atleast from the representation 
in the YARN meetup. I'm going to create a wiki-page listing all these scenarios.

But back to the bug that Junping raised. At this point, we don't have a clear 
path towards running 2.x applications on 3.0.0 clusters. So, our claim of 
rolling-upgrades already working is not accurate.

One of the two options that Junping proposed should be pursued before we close 
the release. I'm in favor of calling out rolling-upgrade support be with-drawn 
or caveated and push for progress instead of blocking the release.

Thanks
+Vinod

> On Dec 12, 2017, at 5:44 PM, Junping Du  wrote:
> 
> Thanks Andrew for pushing new RC for 3.0.0. I was out last week, just get 
> chance to validate new RC now.
> 
> Basically, I found two critical issues with the same rolling upgrade scenario 
> as where HADOOP-15059 get found previously:
> HDFS-12920, we changed value format for some hdfs configurations that old 
> version MR client doesn't understand when fetching these configurations. Some 
> quick workarounds are to add old value (without time unit) in hdfs-site.xml 
> to override new default values but will generate many annoying warnings. I 
> provided my fix suggestions on the JIRA already for more discussion.
> The other one is YARN-7646. After we workaround HDFS-12920, will hit the 
> issue that old version MR AppMaster cannot communicate with new version of 
> YARN RM - could be related to resource profile changes from YARN side but 
> root cause are still in investigation.
> 
> The first issue may not belong to a blocker given we can workaround this 
> without code change. I am not sure if we can workaround 2nd issue so far. If 
> not, we may have to fix this or compromise with withdrawing support of 
> rolling upgrade or calling it a stable release.
> 
> 
> Thanks,
> 
> Junping
> 
> 
> From: Robert Kanter 
> Sent: Tuesday, December 12, 2017 3:10 PM
> To: Arun Suresh
> Cc: Andrew Wang; Lei Xu; Wei-Chiu Chuang; Ajay Kumar; Xiao Chen; Aaron T. 
> Myers; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
> yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC1
> 
> +1 (binding)
> 
> + Downloaded the binary release
> + Deployed on a 3 node cluster on CentOS 7.3
> + Ran some MR jobs, clicked around the UI, etc
> + Ran some CLI commands (yarn logs, etc)
> 
> Good job everyone on Hadoop 3!
> 
> 
> - Robert
> 
> On Tue, Dec 12, 2017 at 1:56 PM, Arun Suresh  wrote:
> 
>> +1 (binding)
>> 
>> - Verified signatures of the source tarball.
>> - built from source - using the docker build environment.
>> - set up a pseudo-distributed test cluster.
>> - ran basic HDFS commands
>> - ran some basic MR jobs
>> 
>> Cheers
>> -Arun
>> 
>> On Tue, Dec 12, 2017 at 1:52 PM, Andrew Wang 
>> wrote:
>> 
>>> Hi everyone,
>>> 
>>> As a reminder, this vote closes tomorrow at 12:31pm, so please give it a
>>> whack if you have time. There are already enough binding +1s to pass this
>>> vote, but it'd be great to get additional validation.
>>> 
>>> Thanks to everyone who's voted thus far!
>>> 
>>> Best,
>>> Andrew
>>> 
>>> 
>>> 
>>> On Tue, Dec 12, 2017 at 11:08 AM, Lei Xu  wrote:
>>> 
 +1 (binding)
 
 * Verified src tarball and bin tarball, verified md5 of each.
 * Build source with -Pdist,native
 * Started a pseudo cluster
 * Run ec -listPolicies / -getPolicy / -setPolicy on /  , and run hdfs
 dfs put/get/cat on "/" with XOR-2-1 policy.
 
 Thanks Andrew for this great effort!
 
 Best,
 
 
 On Tue, Dec 12, 2017 at 9:55 AM, Andrew Wang >> 
 wrote:
> Hi Wei-Chiu,
> 
> The patchprocess directory is left over from the create-release
>>> process,
> and it looks empty to me. We should still file a create-release JIRA
>> to
 fix
> this, but I think this is not a blocker. Would you agree?
> 
> Best,
> Andrew
> 
> On Tue, Dec 12, 2017 at 9:44 AM, Wei-Chiu Chuang <
>> weic...@cloudera.com
 
> wrote:
> 
>> Hi Andrew, thanks the tremendous effort.
>> I found an empty "patchprocess" directory in the source tarball,
>> that
>>> is
>> not there if you clone from github. Any chance you might have some
 leftover
>> trash when you made the tarball?
>> Not wanting to nitpicking, but you might want to double check so we
 don't
>> ship anything private to you in public :)
>> 
>> 
>> 
>> On Tue, Dec 12, 2017 at 7:48 AM, Ajay Kumar <
>>> ajay.ku...@hortonworks.com
> 
>> wrote:
>> 
>>> +1 (non-binding)
>>> Thanks for driving 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

Looked at RC1. Went through my usual check-list. Here's my summary.

+1 (binding) overall

Verification
- [Check] Successful recompilation from source tar-ball
- [Check] Signature verification
- [Check] Generating dist tarballs from source tar-ball
- [Check] Validating the layout of the binary tar-ball
- [Check] Testing
   -- Start NN, DN, RM, NM, JHS, Timeline Service
   -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
   -- Tested CLIs to print nodes, apps etc and also navigated UIs

Few issues as before found during testing, but shouldn't be blockers
 - The previously supported way of being able to use different tar-balls for 
different sub-modules is completely broken - common and HDFS tar.gz are 
completely empty. Will file a ticket.
 - resourcemanager-metrics.out is going into current directory instead of log 
directory. Will file a ticket.

One thing I want to make sure folks agree to. Allen filed a ticket to remove 
yarn-historyserver option per our previous discussion - 
https://issues.apache.org/jira/browse/YARN-7588 
. It isn't done - I am hoping 
this 'incompatible change' can be put in 3.0.1 and not 4.0.

Thanks
+Vinod


> On Dec 8, 2017, at 12:31 PM, Andrew Wang  wrote:
> 
> Hi all,
> 
> Let me start, as always, by thanking the efforts of all the contributors
> who contributed to this release, especially those who jumped on the issues
> found in RC0.
> 
> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
> fixed JIRAs since the previous 3.0.0-beta1 release.
> 
> You can find the artifacts here:
> 
> http://home.apache.org/~wang/3.0.0-RC1/
> 
> I've done the traditional testing of building from the source tarball and
> running a Pi job on a single node cluster. I also verified that the shaded
> jars are not empty.
> 
> Found one issue that create-release (probably due to the mvn deploy change)
> didn't sign the artifacts, but I fixed that by calling mvn one more time.
> Available here:
> 
> https://repository.apache.org/content/repositories/orgapachehadoop-1075/
> 
> This release will run the standard 5 days, closing on Dec 13th at 12:31pm
> Pacific. My +1 to start.
> 
> Best,
> Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Vinod Kumar Vavilapalli]

I couldn't find the release tag for RC1 either - is it just me or has the 
release-process changed?

+Vinod

> On Dec 10, 2017, at 4:31 PM, Sangjin Lee  wrote:
> 
> Hi Andrew,
> 
> Thanks much for your effort! Just to be clear, could you please state the
> git commit id of the RC1 we're voting for?
> 
> Sangjin
> 
> On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang 
> wrote:
> 
>> Hi all,
>> 
>> Let me start, as always, by thanking the efforts of all the contributors
>> who contributed to this release, especially those who jumped on the issues
>> found in RC0.
>> 
>> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
>> fixed JIRAs since the previous 3.0.0-beta1 release.
>> 
>> You can find the artifacts here:
>> 
>> http://home.apache.org/~wang/3.0.0-RC1/
>> 
>> I've done the traditional testing of building from the source tarball and
>> running a Pi job on a single node cluster. I also verified that the shaded
>> jars are not empty.
>> 
>> Found one issue that create-release (probably due to the mvn deploy change)
>> didn't sign the artifacts, but I fixed that by calling mvn one more time.
>> Available here:
>> 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1075/
>> 
>> This release will run the standard 5 days, closing on Dec 13th at 12:31pm
>> Pacific. My +1 to start.
>> 
>> Best,
>> Andrew
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Vinod Kumar Vavilapalli]

Missed this response on the old thread, but closing the loop here..

The incompatibility conundrum with Dot-zeroes did indeed happen, in early 2.x 
releases - multiple times at that. And the downstream projects did raise 
concerns at these unfixable situations.

I wasn't advocating a new formalism, this was more of a lesson taken from real 
life experience that I wanted share with fellow RMs - as IMO the effort was 
worth the value for the releases where I used it.

If RMs of these more recent releases choose to not do this if it is perceived 
that a release won't run into those past issues at all, it's clearly their 
call. It's just that we are bound to potentially make the same mistakes and 
learn the same lesson all over again..

+Vinod

> On Nov 9, 2017, at 9:51 AM, Chris Douglas <cdoug...@apache.org> wrote:
> 
> The labor required for these release formalisms is exceeding their
> value. Our minor releases have more bugs than our patch releases (we
> hope), but every consumer should understand how software versioning
> works. Every device I own has bugs on major OS updates. That doesn't
> imply that every minor release is strictly less stable than a patch
> release, and users need to be warned off it.
> 
> In contrast, we should warn users about features that compromise
> invariants like security or durability, either by design or due to
> their early stage of development. We can't reasonably expect them to
> understand those tradeoffs, since they depend on internal details of
> Hadoop.
> 
> On Wed, Nov 8, 2017 at 5:34 PM, Vinod Kumar Vavilapalli
> <vino...@apache.org <mailto:vino...@apache.org>> wrote:
>> When we tried option (b), we used to make .0 as a GA release, but downstream 
>> projects like Tez, Hive, Spark would come back and find an incompatible 
>> change - and now we were forced into a conundrum - is fixing this 
>> incompatible change itself an incompatibility?
> 
> Every project takes these case-by-case. Most of the time we'll
> accommodate the old semantics- and we try to be explicit where we
> promise compatibility- but this isn't a logic problem, it's a
> practical one. If it's an easy fix to an obscure API, we probably
> won't even hear about it.
> 
>> Long story short, I'd just add to your voting thread and release notes that 
>> 2.9.0 still needs to be tested downstream and so users may want to wait for 
>> subsequent point releases.
> 
> It's uncomfortable to have four active release branches, with 3.1
> coming in early 2018. We all benefit from the shared deployment
> experiences that harden these releases, and fragmentation creates
> incentives to compete for that attention. Rather than tacitly
> scuffling over waning interest in the 2.x series, I'd endorse your
> other thread encouraging consolidation around 3.x.
> 
> To that end, there is no policy or precedent that requires that new
> minor releases be labeled as "alpha". If there is cause to believe
> that 2.9.0 is not ready to release in the stable line, then we
> shouldn't release it. -C
> 
>>> On Nov 8, 2017, at 12:43 AM, Subru Krishnan <su...@apache.org> wrote:
>>> 
>>> We are canceling the RC due to the issue that Rohith/Sunil identified. The
>>> issue was difficult to track down as it only happens when you use IP for ZK
>>> (works fine with host names) and moreover if ZK and RM are co-located on
>>> same machine. We are hopeful to get the fix in tomorrow and roll out RC1.
>>> 
>>> Thanks to everyone for the extensive testing/validation. Hopefully cost to
>>> replicate with RC1 is much lower.
>>> 
>>> -Subru/Arun.
>>> 
>>> On Tue, Nov 7, 2017 at 5:27 PM, Konstantinos Karanasos <kkarana...@gmail.com
>>>> wrote:
>>> 
>>>> +1 from me too.
>>>> 
>>>> Did the following:
>>>> 1) set up a 9-node cluster;
>>>> 2) ran some Gridmix jobs;
>>>> 3) ran (2) after enabling opportunistic containers (used a mix of
>>>> guaranteed and opportunistic containers for each job);
>>>> 4) ran (3) but this time enabling distributed scheduling of opportunistic
>>>> containers.
>>>> 
>>>> All the above worked with no issues.
>>>> 
>>>> Thanks for all the effort guys!
>>>> 
>>>> Konstantinos
>>>> 
>>>> 
>>>> 
>>>> Konstantinos
>>>> 
>>>> On Tue, Nov 7, 2017 at 2:56 PM, Eric Badger <ebad...@oath.com.invalid>
>>>> wrote:
>>>> 
>>>>> +1 (non-binding) pending the issue that Sunil/Rohith pointed out
>>>>> 
>>>>&

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even 
>> work. Not just deprecated in favor of timelineserver as was advertised.
> 
>   This works for me in trunk and the bash code doesn’t appear to have 
> changed in a very long time.  Probably something local to your install.  (I 
> do notice that the deprecation message says “starting” which is awkward when 
> the stop command is given though.)  Also: is the deprecation message even 
> true at this point?


Sorry, I mischaracterized the problem.

The real issue is that I cannot use this command line when the MapReduce 
JobHistoryServer is already started on the same machine.

~/tmp/yarn$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver
WARNING: Use of this script to start YARN daemons is deprecated.
WARNING: Attempting to execute replacement "yarn --daemon start" instead.
DEPRECATED: Use of this command to start the timeline server is deprecated.
Instead use the timelineserver command for it.
Starting the History Server anyway...
historyserver is running as process 86156.  Stop it first.

So, it looks like in shell-scripts, there can ever be only one daemon of a 
given name, irrespective of which daemon scripts are invoked.

We need to figure out two things here
 (a) The behavior of this command. Clearly, it will conflict with the MapReduce 
JHS - only one of them can be started on the same node.
 (b) We need to figure out if this V1 TimelineService should even be support 
given ATSv2.

@Vrushani / @Rohith / @Varun Saxena et.al, if you are watching, please comment 
on (b).

Thanks
+Vinod

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>>> - Cannot enable new UI in YARN because it is under a non-default
>>> compilation flag. It should be on by default.
>>> 
>> 
>> The yarn-ui profile has always been off by default, AFAIK. It's documented
>> to turn it on in BUILDING.txt for release builds, and we do it in
>> create-release.
>> 
>> IMO not a blocker. I think it's also more of a dev question (do we want to
>> do this on every YARN build?) than a release one.
> 
>   -1 on making yarn-ui always build.
> 
>   For what is effectively an optional component (the old UI is still 
> there), it’s heavy dependency requirements make it a special burden outside 
> of the Docker container.


Thanks for the background on this.

I got schooled offline too about the heaviness of the dependencies.


> If it can be changed such that it either always downloads the necessary bits 
> (regardless of the OS/chipset!) and/or doesn’t kill the maven build if those 
> bits can’t be found  (i.e., truly optional), then I’d be less opposed.  (and, 
> actually, quite pleased because then the docker image build would be 
> significantly faster.)


This is a good idea (to the extend I understand your proposal). Long term, we'd 
like YARN UI2 to replace current UI, so it shouldn't be optionally build and 
hence the build should be fast. Let me track this separately as a non-blocker.

Thanks
+Vinod
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>> - One decommissioned node in YARN ResourceManager UI always appears to
>> start with, even when there are no NodeManagers that are started yet:
>> Info :-1, DECOMMISSIONED, null rack. It shows up only in the UI though,
>> not in the CLI node -list
>> 
> 
> Is this a blocker? Could we get a JIRA?


I just cross verified this with others - looks like an issue with my 
environment - not a blocker, ignore.

+Vinod
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Thanks for all the push, Andrew!

Looking at the RC. Went through my usual check-list. Here's my summary. Will 
cast my final vote after comparing and validating my findings with others.

Verification

 - [Check] Successful recompilation from source tar-ball
 - [Check] Signature verification
 - [Check] Generating dist tarballs from source tar-ball
 - [Check] Testing
-- Start NN, DN, RM, NM, JHS, Timeline Service
-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
-- Tested CLIs to print nodes, apps etc and also navigated UIs

Issues found during testing

Major
 - The previously supported way of being able to use different tar-balls for 
different sub-modules is completely broken - common and HDFS tar.gz are 
completely empty.
 - Cannot enable new UI in YARN because it is under a non-default compilation 
flag. It should be on by default.
 - One decommissioned node in YARN ResourceManager UI always appears to start 
with, even when there are no NodeManagers that are started yet:  Info :-1, 
DECOMMISSIONED, null rack. It shows up only in the UI though, not in the CLI 
node -list

Minor
 - resourcemanager-metrics.out is going into current directory instead of log 
directory
 - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even work. 
Not just deprecated in favor of timelineserver as was advertised.
 - Spurious warnings on CLI
17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml not 
found
17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find 
'resource-types.xml'.

Side notes

 - When did we stop putting CHANGES files into the source artifacts?
 - Even after "mvn install"ing once, shading is repeated again and again for 
every new 'mvn install' even though there are no source changes - we should see 
how this can be avoided.
 - Compatibility notes
-- NM's env list is curtailed unlike in 2.x (For e.g, HADOOP_MAPRED_HOME is 
not automatically inherited. Correct behavior)
-- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar into 
hadoop-mapreduce-client-jobclient-3.0.0-tests.jar

Thanks
+Vinod

> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Compilation passed for me. Using jdk1.8.0_40.jdk.

+Vinod

> On Nov 20, 2017, at 4:16 PM, Wei-Chiu Chuang <weic...@cloudera.com> wrote:
> 
> @vinod
> I followed your command but I could not reproduce your problem.
> 
> [weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-common-project/hadoop-c
> ommon/target/hadoop-common-3.0.0.tar.gz
> -rw-rw-r-- 1 weichiu weichiu 37052439 Nov 20 21:59
> hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz
> [weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-hdfs-project/hadoop-hdf
> s/target/hadoop-hdfs-3.0.0.tar.gz
> -rw-rw-r-- 1 weichiu weichiu 29044067 Nov 20 22:00
> hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
> 
> During compilation I found the following error with a Java 1.8.0_5 JDK:
> 
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven
> -compiler-plugin:3.1:testCompile (default-testCompile) on project
> hadoop-aws: Compilation failure: Compilation failure:
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[45,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[69,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[94,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[120,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> <T,E>intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> 
> And then I realized Ray filed HADOOP-14900
> <https://issues.apache.org/jira/browse/HADOOP-14900> for the same
> issue. This problem is not reproducible with a more recent JDK 8, such as
> 1.8.0_151
> Maybe it would be a good idea to name a list of JDKs that are known to be
> buggy. Can we get this documented somewhere? I don't consider it a blocker
> so a release note in a later release or a wiki entry should be good enough.
> 
> On Mon, Nov 20, 2017 at 12:58 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
> 
>> Quick question.
>> 
>> I used to be able (in 2.x line) to create dist tarballs (mvn clean install
>> -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being
>> voted on (hadoop-3.0.0-src.tar.gz).
>> 
>> The idea is to install HDFS, YARN, MR separately in separate
>> root-directories from the generated individual dist tarballs.
>> 
>> But now I see that HDFS and common dist tarballs are empty
>> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39
>> ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz -
>> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40
>> ./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
>> 
>> But YARN and MR are fine
>> -rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41
>> ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
>> -rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41
>> ./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz
>> 
>> Is it just me? Or is this broken?
>> 
>> Thanks
>> +Vinod
>> 
>>> On Nov 14, 2017, at 1:34 PM, Andrew Wang <andrew.w...@cloude

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Quick question.

I used to be able (in 2.x line) to create dist tarballs (mvn clean install 
-Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being voted 
on (hadoop-3.0.0-src.tar.gz).

The idea is to install HDFS, YARN, MR separately in separate root-directories 
from the generated individual dist tarballs.

But now I see that HDFS and common dist tarballs are empty
-rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39 
./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz - 
-rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40 
./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz

But YARN and MR are fine
-rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41 
./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
-rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41 
./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz

Is it just me? Or is this broken?

Thanks
+Vinod

> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

I'd definitely extend it for a few more days. I only see 3 binding +1s so far - 
not a great number to brag about on our first major release in years.

Also going to nudge folks into voting.

+Vinod

> On Nov 17, 2017, at 3:26 PM, Andrew Wang  wrote:
> 
> Hi Arpit,
> 
> I agree the timing is not great here, but extending it to meaningfully
> avoid the holidays would mean extending it an extra week (e.g. to the
> 29th). We've been coordinating with ASF PR for that Tuesday, so I'd really,
> really like to get the RC out before then.
> 
> In terms of downstream testing, we've done extensive integration testing
> with downstreams via the alphas and betas, and we have continuous
> integration running at Cloudera against branch-3.0. Because of this, I have
> more confidence in our integration for 3.0.0 than most Hadoop releases.
> 
> Is it meaningful to extend to say, the 21st, which provides for a full week
> of voting?
> 
> Best,
> Andrew
> 
> On Fri, Nov 17, 2017 at 1:27 PM, Arpit Agarwal 
> wrote:
> 
>> Hi Andrew,
>> 
>> Thank you for your hard work in getting us to this step. This is our first
>> major GA release in many years.
>> 
>> I feel a 5-day vote window ending over the weekend before thanksgiving may
>> not provide sufficient time to evaluate this RC especially for downstream
>> components.
>> 
>> Would you please consider extending the voting deadline until a few days
>> after the thanksgiving holiday? It would be a courtesy to our broader
>> community and I see no harm in giving everyone a few days to evaluate it
>> more thoroughly.
>> 
>> On a lighter note, your deadline is also 4 minutes short of the required 5
>> days. :)
>> 
>> Regards,
>> Arpit
>> 
>> 
>> 
>> On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:
>> 
>>Hi folks,
>> 
>>Thanks as always to the many, many contributors who helped with this
>>release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>>available here:
>> 
>>http://people.apache.org/~wang/3.0.0-RC0/
>> 
>>This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>> 
>>3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>>additions include the merge of YARN resource types, API-based
>> configuration
>>of the CapacityScheduler, and HDFS router-based federation.
>> 
>>I've done my traditional testing with a pseudo cluster and a Pi job.
>> My +1
>>to start.
>> 
>>Best,
>>Andrew
>> 
>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Apache YARN committers/contribu­t­ors meetup #5

[Vinod Kumar Vavilapalli]

Thanks for volunteering to host, Daniel!

The agenda looks good. But we could also go a completely orthogonal way and do 
a bug / review bash too. Up for either of these directions. Or a mix.

Thanks
+Vinod

> On Nov 17, 2017, at 9:22 AM, Daniel Templeton <dan...@cloudera.com> wrote:
> 
> I will close the poll this afternoon.  Right now the overwhelming preference 
> is for Dec 6th.  Remote folks note that we will have a dial-in option, most 
> likely via Hangout.
> 
> Daniel
> 
> On 11/14/17 10:39 AM, Daniel Templeton wrote:
>> I was hoping for some signs of life before posting a Doodle poll, but I 
>> guess we can head off the flurry of votes on this thread with a poll now:
>> 
>> https://doodle.com/poll/mmzmsyvdk9n5pudc
>> 
>> Everyone, please fill out the poll, even if you can't attend either of the 
>> dates.  Let's try to make a call on a date this week.
>> 
>> Daniel
>> 
>> On 11/13/17 5:43 PM, Karthik Kambatla wrote:
>>> Thanks for hosting this.
>>> 
>>> Should people respond to this email for date preferences or do you want to 
>>> use something like doodle?
>>> 
>>> 
>>> 
>>> On Sat, Nov 11, 2017 at 8:59 AM Daniel Templeton <dan...@cloudera.com 
>>> <mailto:dan...@cloudera.com>> wrote:
>>> 
>>>Sounds like there are no other volunteers, so we're happy to host
>>>here
>>>at Cloudera.  How about November 29th or December 6th in the
>>>afternoon?
>>>Suggested topics would include:
>>> 
>>>3.0 status/update/celebration
>>>3.1 status, timing, and plans
>>>What we're doing with the 2.x branch going forward
>>>Docker status/plans
>>>Other?
>>> 
>>>Thanks!
>>>Daniel
>>> 
>>>On 10/20/17 1:54 PM, Daniel Templeton wrote:
>>>> Seems to me like we're due for a YARN contributors meetup.  Anyone
>>>> want to volunteer to host?  I'd be happy to handle the
>>>logistics and
>>>> host here at Cloudera, but I don't want to take the opportunity
>>>away
>>>> from another company. :)
>>>>
>>>> Daniel
>>>>
>>>> On 10/28/16 3:27 PM, Vinod Kumar Vavilapalli wrote:
>>>>> Thanks to everyone who joined this meetup!
>>>>>
>>>>> We had quite a blast both in the western hemisphere and from
>>>what I
>>>>> hear in the IST timezone too.
>>>>>
>>>>> Overall, stats
>>>>>
>>>>>   - PST
>>>>>  — Started at 269 patch-available tickets and got it down
>>>to 170
>>>>> - a mix of commits, reviews + updates, closing invalid /
>>>>> not-applicable JIRAs
>>>>>  — Working notes:
>>>>>
>>>
>>> https://docs.google.com/spreadsheets/d/1kPKsm3VSnkLU107t-CL05RQ9xQxc6kN-h6o9bDrheaY/edit#gid=2076540402
>>>>>
>>>>>   - IST: (Notes from Sunil)
>>>>>  — 19 patch commits, 11 added / rebased patches, and more
>>>commits
>>>>> on the way waiting for Jenkins
>>>>>  — Working notes:
>>>>>
>>>
>>> https://docs.google.com/spreadsheets/d/1EVga79x-sxrfxWoe3o_ZkyLgHbrzHJqAU4hq6qabW_k/edit?ts=5811eb51#gid=0
>>>>>
>>>>> Special thanks
>>>>>   - To Subru for sponsoring the event, logistics and a great
>>>lunch!
>>>>>   - To Sunil for taking the initiative, organizing and running the
>>>>> contributors’ meetup in India!
>>>>>
>>>>> We are thinking of doing this at a regular cadence but for a
>>>smaller
>>>>> duration than a full-day.
>>>>>
>>>>> Thanks
>>>>> +Vinod
>>>>>
>>>>>> On Oct 19, 2016, at 11:08 AM, Subru Krishnan
>>><su...@apache.org <mailto:su...@apache.org>> wrote:
>>>>>>
>>>>>> Folks,
>>>>>>
>>>>>> Hope everyone's is doing great.
>>>>>>
>>>>>> We are putting in one full day (5-6 hours) for a YARN review
>>>/ commit
>>>>>> marathon on *next Thursday, 27th Oct*.
>>>>>>

Re: [DISCUSS] A final minor release off branch-2?

[Vinod Kumar Vavilapalli]

Thanks for your comments, Zheng. Replies inline.


> On the other hand, I've discussed with quite a few 3.0 potential users, it 
> looks like most of them are interested in the erasure coding feature and a 
> major scenario for that is to back up their large volume of data to save 
> storage cost. They might run analytics workload using Hive, Spark, Impala and 
> Kylin on the new cluster based on the version, but it's not a must at the 
> first time. They understand there might be some gaps so they'd migrate their 
> workloads incrementally. For the major analytics workload, we've performed 
> lots of benchmark and integration tests as well as other sides I believe, we 
> did find some issues but they should be fixed in downstream projects. I 
> thought the release of GA will accelerate the progress and expose the issues 
> if any. We couldn't wait for it being matured. There isn't perfectness.


3.0 is a GA release from the Apache Hadoop community. So, we cannot assume that 
all usages in the short term are *only* going to be for storage optimization 
features and only on dedicated clusters. We have to make sure that the 
workloads can be migrated right now and/or that existing clusters can be 
upgraded in-place. If not, we shouldn't be calling it GA.


> This sounds a good consideration. I'm thinking if I'm a Hadoop user, for 
> example, I'm using 2.7.4 or 2.8.2 or whatever 2.x version, would I first 
> upgrade to this bridging release then use the bridge support to upgrade to 
> 3.x version? I'm not sure. On the other hand, I might tend to look for some 
> guides or supports in 3.x docs about how to upgrade from 2.7 to 3.x. 



Arun Suresh also asked this same question earlier. I think this will really 
depend on what we discover as part of the migration and user-acceptance 
testing. If we don't find major issues, you are right, folks can jump directly 
from one of 2.7, 2.8 or 2.9 to 3.0.



> Frankly speaking, working on some bridging release not targeting any feature 
> isn't so attractive to me as a contributor. Overall, the final minor release 
> off branch-2 is good, we should also give 3.x more time to evolve and mature, 
> therefore it looks to me we would have to work on two release lines meanwhile 
> for some time. I'd like option C), and suggest we focus on the recent 
> releases.



Answering this question is also one of the goals of my starting this thread. 
Collectively we need to conclude if we are okay or not okay with no longer 
putting any new feature work in general on the 2.x line after 2.9.0 release and 
move over our focus into 3.0.


Thanks
+Vinod

> -Original Message-
> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] 
> Sent: Tuesday, November 07, 2017 9:43 AM
> To: Andrew Wang <andrew.w...@cloudera.com>
> Cc: Arun Suresh <asur...@apache.org>; common-...@hadoop.apache.org; 
> yarn-dev@hadoop.apache.org; Hdfs-dev <hdfs-...@hadoop.apache.org>; 
> mapreduce-...@hadoop.apache.org
> Subject: Re: [DISCUSS] A final minor release off branch-2?
> 
> The main goal of the bridging release is to ease transition on stuff that is 
> guaranteed to be broken.
> 
> Of the top of my head, one of the biggest areas is application compatibility. 
> When folks move from 2.x to 3.x, are their apps binary compatible? Source 
> compatible? Or need changes?
> 
> In 1.x -> 2.x upgrade, we did a bunch of work to atleast make old apps be 
> source compatible. This means relooking at the API compatibility in 3.x and 
> their impact of migrating applications. We will have to revist and 
> un-deprecate old APIs, un-delete old APIs and write documentation on how apps 
> can be migrated.
> 
> Most of this work will be in 3.x line. The bridging release on the other hand 
> will have deprecation for APIs that cannot be undeleted. This may be already 
> have been done in many places. But we need to make sure and fill gaps if any.
> 
> Other areas that I can recall from the old days
> - Config migration: Many configs are deprecated or deleted. We need 
> documentation to help folks to move. We also need deprecations in the 
> bridging release for configs that cannot be undeleted.
> - You mentioned rolling-upgrades: It will be good to exactly outline the type 
> of testing. For e.g., the rolling-upgrades orchestration order has direct 
> implication on the testing done.
> - Story for downgrades?
> - Copying data between 2.x clusters and 3.x clusters: Does this work already? 
> Is it broken anywhere that we cannot fix? Do we need bridging features for 
> this work?
> 
> +Vinod
> 
>> On Nov 6, 2017, at 12:49 PM, Andrew Wang <andrew.w...@cloudera.com> wrote:
>> 
>> What are the known gaps that need bridging between 2.x and 3.x?
>> 
>> From an HDFS perspective,

Re: [DISCUSS] A final minor release off branch-2?

[Vinod Kumar Vavilapalli]

The main goal of the bridging release is to ease transition on stuff that is 
guaranteed to be broken.

Of the top of my head, one of the biggest areas is application compatibility. 
When folks move from 2.x to 3.x, are their apps binary compatible? Source 
compatible? Or need changes?

In 1.x -> 2.x upgrade, we did a bunch of work to atleast make old apps be 
source compatible. This means relooking at the API compatibility in 3.x and 
their impact of migrating applications. We will have to revist and un-deprecate 
old APIs, un-delete old APIs and write documentation on how apps can be 
migrated.

Most of this work will be in 3.x line. The bridging release on the other hand 
will have deprecation for APIs that cannot be undeleted. This may be already 
have been done in many places. But we need to make sure and fill gaps if any.

Other areas that I can recall from the old days
 - Config migration: Many configs are deprecated or deleted. We need 
documentation to help folks to move. We also need deprecations in the bridging 
release for configs that cannot be undeleted.
 - You mentioned rolling-upgrades: It will be good to exactly outline the type 
of testing. For e.g., the rolling-upgrades orchestration order has direct 
implication on the testing done.
 - Story for downgrades?
 - Copying data between 2.x clusters and 3.x clusters: Does this work already? 
Is it broken anywhere that we cannot fix? Do we need bridging features for this 
work?

+Vinod

> On Nov 6, 2017, at 12:49 PM, Andrew Wang <andrew.w...@cloudera.com> wrote:
> 
> What are the known gaps that need bridging between 2.x and 3.x?
> 
> From an HDFS perspective, we've tested wire compat, rolling upgrade, and
> rollback.
> 
> From a YARN perspective, we've tested wire compat and rolling upgrade. Arun
> just mentioned an NM rollback issue that I'm not familiar with.
> 
> Anything else? External to this discussion, these should be documented as
> known issues for 3.0.
> 
> Best.
> Andrew
> 
> On Sun, Nov 5, 2017 at 1:46 PM, Arun Suresh <asur...@apache.org> wrote:
> 
>> Thanks for starting this discussion VInod.
>> 
>> I agree (C) is a bad idea.
>> I would prefer (A) given that ATM, branch-2 is still very close to
>> branch-2.9 - and it is a good time to make a collective decision to lock
>> down commits to branch-2.
>> 
>> I think we should also clearly define what the 'bridging' release should
>> be.
>> I assume it means the following:
>> * Any 2.x user wanting to move to 3.x must first upgrade to the bridging
>> release first and then upgrade to the 3.x release.
>> * With regard to state store upgrades (at least NM state stores) the
>> bridging state stores should be aware of all new 3.x keys so the implicit
>> assumption would be that a user can only rollback from the 3.x release to
>> the bridging release and not to the old 2.x release.
>> * Use the opportunity to clean up deprecated API ?
>> * Do we even want to consider a separate bridging release for 2.7, 2.8 an
>> 2.9 lines ?
>> 
>> Cheers
>> -Arun
>> 
>> On Fri, Nov 3, 2017 at 5:07 PM, Vinod Kumar Vavilapalli <
>> vino...@apache.org>
>> wrote:
>> 
>>> Hi all,
>>> 
>>> With 3.0.0 GA around the corner (tx for the push, Andrew!), 2.9.0 RC out
>>> (tx Arun / Subru!) and 2.8.2 (tx Junping!), I think it's high time we
>> have
>>> a discussion on how we manage our developmental bandwidth between 2.x
>> line
>>> and 3.x lines.
>>> 
>>> Once 3.0 GA goes out, we will have two parallel and major release lines.
>>> The last time we were in this situation was back when we did 1.x -> 2.x
>>> jump.
>>> 
>>> The parallel releases implies overhead of decisions, branch-merges and
>>> back-ports. Right now we already do backports for 2.7.5, 2.8.2, 2.9.1,
>>> 3.0.1 and potentially a 3.1.0 in a few months after 3.0.0 GA. And many of
>>> these lines - for e.g 2.8, 2.9 - are going to be used for a while at a
>>> bunch of large sites! At the same time, our users won't migrate to 3.0 GA
>>> overnight - so we do have to support two parallel lines.
>>> 
>>> I propose we start thinking of the fate of branch-2. The idea is to have
>>> one final release that helps our users migrate from 2.x to 3.x. This
>>> includes any changes on the older line to bridge compatibility issues,
>>> upgrade issues, layout changes, tooling etc.
>>> 
>>> We have a few options I think
>>> (A)
>>>-- Make 2.9.x the last minor release off branch-2
>>>-- Have a maintenance release that bridges 2.9 to 3.x
>>>-- Continue to make more ma

Re: [VOTE] Merge yarn-native-services branch into trunk

[Vinod Kumar Vavilapalli]

Congratulations to all the contributors involved, this is a great step forward!

+Vinod

> On Nov 6, 2017, at 2:40 PM, Jian He <j...@hortonworks.com> wrote:
> 
> Okay, I just merged the branch to trunk (108 commits in total !)
> Again, thanks for all who contributed to this feature!
> 
> Jian
> 
> On Nov 6, 2017, at 1:26 PM, Jian He 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
> 
> Here’s +1 from myself.
> The vote passes with 7 (+1) bindings and 2 (+1) non-bindings.
> 
> Thanks for all who voted. I’ll merge to trunk by the end of today.
> 
> Jian
> 
> On Nov 6, 2017, at 8:38 AM, Billie Rinaldi 
> <billie.rina...@gmail.com<mailto:billie.rina...@gmail.com>> wrote:
> 
> +1 (binding)
> 
> On Mon, Oct 30, 2017 at 1:06 PM, Jian He 
> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
> Hi All,
> 
> I would like to restart the vote for merging yarn-native-services to trunk.
> Since last vote, we have been working on several issues in documentation, 
> DNS, CLI modifications etc. We believe now the feature is in a much better 
> shape.
> 
> Some back ground:
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate 
> existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to 
> deploy a service via a simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS service 
> to enable users to discover services deployed on YARN via standard DNS lookup
> - YARN-6419[4]. UI support for native-services on the new YARN UI
> All these new services are optional and are sitting outside of the existing 
> system, and have no impact on existing system if disabled.
> 
> Special thanks to a team of folks who worked hard towards this: Billie 
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K 
> S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible without 
> their ideas and hard work.
> Also thanks Allen for some review and verifications.
> 
> Thanks,
> Jian
> 
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
> 
> 
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[DISCUSS] A final minor release off branch-2?

[Vinod Kumar Vavilapalli]

Hi all,

With 3.0.0 GA around the corner (tx for the push, Andrew!), 2.9.0 RC out (tx 
Arun / Subru!) and 2.8.2 (tx Junping!), I think it's high time we have a 
discussion on how we manage our developmental bandwidth between 2.x line and 
3.x lines.

Once 3.0 GA goes out, we will have two parallel and major release lines. The 
last time we were in this situation was back when we did 1.x -> 2.x jump.

The parallel releases implies overhead of decisions, branch-merges and 
back-ports. Right now we already do backports for 2.7.5, 2.8.2, 2.9.1, 3.0.1 
and potentially a 3.1.0 in a few months after 3.0.0 GA. And many of these lines 
- for e.g 2.8, 2.9 - are going to be used for a while at a bunch of large 
sites! At the same time, our users won't migrate to 3.0 GA overnight - so we do 
have to support two parallel lines.

I propose we start thinking of the fate of branch-2. The idea is to have one 
final release that helps our users migrate from 2.x to 3.x. This includes any 
changes on the older line to bridge compatibility issues, upgrade issues, 
layout changes, tooling etc.

We have a few options I think
 (A)
-- Make 2.9.x the last minor release off branch-2
-- Have a maintenance release that bridges 2.9 to 3.x
-- Continue to make more maintenance releases on 2.8 and 2.9 as necessary
-- All new features obviously only go into the 3.x line as no features can 
go into the maint line.

 (B)
-- Create a new 2.10 release which doesn't have any new features, but as a 
bridging release
-- Continue to make more maintenance releases on 2.8, 2.9 and 2.10 as 
necessary
-- All new features, other than the bridging changes, go into the 3.x line

 (C)
-- Continue making branch-2 releases and postpone this discussion for later

I'm leaning towards (A) or to a lesser extent (B). Willing to hear otherwise.

Now, this obviously doesn't mean blocking of any more minor releases on 
branch-2. Obviously, any interested committer / PMC can roll up his/her 
sleeves, create a release plan and release, but we all need to acknowledge that 
versions are not cheap and figure out how the community bandwidth is split 
overall.

Thanks
+Vinod
PS: The proposal is obviously not to force everyone to go in one direction but 
more of a nudging the community to figure out if we can focus a major part of 
of our bandwidth on one line. I had a similar concern when we were doing 2.8 
and 3.0 in parallel, but the impending possibility of spreading too thin is 
much worse IMO.
PPS: (C) is a bad choice. With 2.8 and 2.9 we are already seeing user adoption 
splintering between two lines. With 2.10, 2.11 etc coexisting with 3.0, 3.1 
etc, we will revisit the mad phase years ago when we had 0.20.x, 0.20-security 
coexisting with 0.21, 0.22 etc.

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Vinod Kumar Vavilapalli]

Arun / Subru,

Thanks for the great work!

Few quick comments
 - Can you cleanup the RC folder to only have tar.gz and src.tar.gz and their 
signatures and delete everything else? So that it's easy to pick up the 
important bits for the voters. For e.g, like this 
http://people.apache.org/~vinodkv/hadoop-2.8.1-RC3/ 

 - Can you put the generated CHANGES.html and releasenotes.html instead of the 
md files, for quicker perusal?

Thanks
+Vinod

> On Nov 3, 2017, at 3:50 PM, Arun Suresh  wrote:
> 
> Hi folks,
> 
> Apache Hadoop 2.9.0 is the first stable release of Hadoop 2.9 line and
> will be the latest stable/production release for Apache Hadoop - it
> includes 30 New Features with 500+ subtasks, 407 Improvements, 787 Bug
> fixes new fixed issues since 2.8.2 .
> 
>  More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
> 
>  New RC is available at:
> http://home.apache.org/~asuresh/hadoop-2.9.0-RC0/
> 
>  The RC tag in git is: release-2.9.0-RC0, and the latest commit id is:
> 6697f0c18b12f1bdb99cbdf81394091f4fef1f0a
> 
>  The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1065/
> *
> 
>  Please try the release and vote; the vote will run for the usual 5
> days, ending on 11/10/2017 4pm PST time.
> 
> Thanks,
> 
> Arun/Subru

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Vinod Kumar Vavilapalli]

>   At a minimum, it should at least be using it’s own maven module for a 
> lot of the bits that generates it’s own maven jars so that we can split this 
> functionality up at build/test time.


I expected this to be the case, but looks like it isn't.

There's lot of value in splitting the HDFS code into smaller modules. 
Definitely newer code like Ozone.

When we did this for YARN, initially there were concerns about module 
proliferation, but looking back, my observations have been that it has done us 
far more good than expected. Starting with the fact that we had clients and 
servers modularized independently, as well as servers from other servers, with 
far cleaner contracts than what we had in Hadoop 1 world.

Thanks
+Vinod

Re: [DISCUSS] Branches and versions for Hadoop 3

[Vinod Kumar Vavilapalli]

+1 to Andrew’s proposal for 3.x releases.

We had fairly elaborate threads on this branching & compatibility topic before. 
One of them’s here: [1]

+1 to what Jason said.
 (a) Incompatible changes are not to be treated lightly.  We need to stop 
breaking stuff and ‘just dump it on trunk'.
 (b) Major versions are expensive. We should hesitate before asking our users 
to move from 2.0 to 3.0 or 3.0 to 4.0 (with incompatible changes) *without* any 
other major value proposition.

Some of the incompatible changes can clear wait while others cannot and so may 
mandate a major release. What are the some of the common types of incompatible 
changes?
 - Renaming APIs, removing deprecated APIs, renaming configuration properties, 
changing the default value of a configuration, changing shell output / logging 
etc:
— Today, we do this on trunk even though the actual effort involved is very 
minimal compared to the overhead it forces in maintaining incompatible trunk.
 - Dependency library updates - updating guava, protobuf etc in Hadoop breaks 
upstreaming applications. I am assuming Classpath Isolation [2] is still a 
blocker for 3.0 GA.
 - JDK upgrades: We tried two different ways with JDK 7 and JDK 8, we need a 
formal policy on this.

If we can managing the above common breaking changes, we can cause less pain to 
our end users.

Here’s what we can do for 3.x / 4.x specifically.
 - Stay on trunk based 3.x releases
 - Avoid all incompatible changes as much as possible
 - If we run into a bunch of minor incompatible changes that have be done, we 
either (a) make the incompatible behavior optional or (b) just park them say 
with an parked-incompatible-change label if making it optional is not possible
 - We create a 4.0 only when (a) we hit the first major incompatible change 
because a major next-step for Hadoop needs it (for e.g. Erasure Coding), and/or 
(b) the number of parked incompatible changes passes a certain threshold. 
Unlike Jason, I don’t see the threshold to be 1 for cases that don’t fit (1).

References
 [1] Looking to a Hadoop 3 release: 
http://markmail.org/thread/2daldggjaeewdmdf#query:+page:1+mid:m6x73t6srlchywsn+state:results
 

 [2] Classpath isolation for downstream client: 
https://issues.apache.org/jira/browse/HADOOP-11656 


Thanks
+Vinod

> On Aug 25, 2017, at 1:23 PM, Jason Lowe  wrote:
> 
> Allen Wittenauer wrote:
> 
> 
>> Doesn't this place an undue burden on the contributor with the first
>> incompatible patch to prove worthiness?  What happens if it is decided that
>> it's not good enough?
> 
> 
> It is a burden for that first, "this can't go anywhere else but 4.x"
> change, but arguably that should not be a change done lightly anyway.  (Or
> any other backwards-incompatible change for that matter.)  If it's worth
> committing then I think it's perfectly reasonable to send out the dev
> announce that there's reason for trunk to diverge from 3.x, cut branch-3,
> and move on.  This is no different than Andrew's recent announcement that
> there's now a need for separating trunk and the 3.0 line based on what's
> about to go in.
> 
> I do not think it makes sense to pay for the maintenance overhead of two
> nearly-identical lines with no backwards-incompatible changes between them
> until we have the need.  Otherwise if past trunk behavior is any
> indication, it ends up mostly enabling people to commit to just trunk,
> forgetting that the thing they are committing is perfectly valid for
> branch-3.  If we can agree that trunk and branch-3 should be equivalent
> until an incompatible change goes into trunk, why pay for the commit
> overhead and potential for accidentally missed commits until it is really
> necessary?
> 
> How many will it take before the dam will break?  Or is there a timeline
>> going to be given before trunk gets set to 4.x?
> 
> 
> I think the threshold count for the dam should be 1.  As soon as we have a
> JIRA that needs to be committed to move the project forward and we cannot
> ship it in a 3.x release then we create branch-3 and move trunk to 4.x.
> As for a timeline going to 4.x, again I don't see it so much as a "baking
> period" as a "when we need it" criteria.  If we need it in a week then we
> should cut it in a week.  Or a year then a year.  It all depends upon when
> that 4.x-only change is ready to go in.
> 
> Given the number of committers that openly ignore discussions like this,
>> who is going to verify that incompatible changes don't get in?
>> 
> 
> The same entities who are verifying other bugs don't get in, i.e.: the
> committers and the Hadoop QA bot running the tests.  Yes, I know that means
> it's inevitable that compatibility breakages will happen, and we can and
> should improve the automation around compatibility testing when possible.
> But I don't think there's a magic 

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

> From a release management perspective, it's *extremely* reasonable to block 
> the inclusion of new features a month from the planned release date. A 
> typical software development lifecycle includes weeks of feature freeze and 
> weeks of code freeze. It is no knock on any developer or any feature to say 
> that we should not include something in 3.0.0.


We have never followed the ‘typical' lifecycle that I am guessing you are 
referring to. If we are, you'll need to publish some of the following: a 
feature freeze date, blockers-criticals-only-from-now date, testing-finish 
date, documentation-finish date, final release date and so on.

What we do with Apache releases typically is instead we say ‘this' is roughly 
when we want to release, and roughly what features must land and let the rest 
figure out itself.

Neither is right or wrong. If we want to change the process, we should 
communicate as such.

Proposing a feature freeze date on the fly is only going to confuse people.


> I've been very open and clear about the goals, schedule, and scope of 3.0.0 
> over the last year plus. The point of the extended alpha process was to get 
> all our features in during alpha, and the alpha merge window has been open 
> for a year. I'm unmoved by arguments about how long a feature has been worked 
> on. None of these were not part of the original 3.0.0 scope, and our users 
> have been waiting even longer for big-ticket 3.0 items like JDK8 and HDFS EC 
> that were part of the discussed scope.


Except our schedule is so fluid (not due to the release management process to 
be fair) that it is hard for folks to plan their features. IIRC, our schedule 
was a GA release beginning of this year. Again, this is not a critique of 3.0 
release process - I have myself done enough releases to know that sticking to a 
date and herding the crowd has been an extremely hard job.

The only way out of this is get 3.0 out and stick to 1-2 month minor releases. 
May be we start that by planning a 3.1 right now and push one in January 
assuming 3.0 GA happens in November. Cadence is a habit.


> I see that two VOTEs have gone out since I was out. I still plan to follow 
> the proposal in my original email. This means I'll cut branch-3 and 
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open up 
> development for Hadoop 3.1.0 and 4.0.0.
> 
> I'm reaching out to the lead contributor of each of these features 
> individually to discuss. We need to close on this quickly, and email is too 
> low bandwidth at this stage.


My only concern in all of this is if some of these branch contributors decide 
that they don’t want and then proceed to having a parallel release and cause 
pains for everyone. This is what I tried avoiding parallel 2.9 and 2.10 offline 
and that’s what I am trying to do here. For now, I don’t have a horse in this 
race - that’s between you and the branch-contributors in question for now. If 
you can reach an agreement, we are all good for it.

+Vinod



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Agreed. I was very clearly not advocating for rushing in features. If you have 
followed my past emails, I have only strongly advocated features be worked in 
branches and get merged when they are in a reasonable state.

Each branch contributor group should look at their readiness and merge stuff in 
assuming that the branch reached a satisfactory state. That’s it.

From release management perspective, blocking features just because we are a 
month close to the deadline is not reasonable. Let the branch contributors 
rationalize, make this decision and the rest of us can support them in making 
the decision.

+Vinod

> At this point, there have been three planned alphas from September 2016 until 
> July 2017 to "get in features".  While a couple of upcoming features are "a 
> few weeks" away, I think all of us are aware how predictable software 
> development schedules can be.  I think we can also all agree that rushing 
> just to meet a release deadline isn't the best practice when it comes to 
> software development either.
> 
> Andrew has been very clear about his goals at each step and I think Wangda's 
> willingness to not rush in resource types was an appropriate response.  I'm 
> sympathetic to the goals of getting in a feature for 3.0, but it might be a 
> good idea for each project that is a "few weeks away" to seriously look at 
> the readiness compared to the features which have been testing for 6+ months 
> already.
> 
> -Ray

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Vinod Kumar Vavilapalli]

Such a great community effort - hats off, team!

Thanks
+Vinod

> On Aug 21, 2017, at 11:32 PM, Vrushali Channapattan <vrushalic2...@gmail.com> 
> wrote:
> 
> Hi folks,
> 
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
> 
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
> 
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
> 
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
> 
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
> 
> There are a total of 82 subtasks that were completed as part of this effort.
> 
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
> 
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> 
> Regards,
> Vrushali
> 
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Steve,

You can be strict & ruthless about the timelines. Anything that doesn’t get in 
by mid-September, as was originally planned, can move to the next release - 
whether it is feature work on branches or feature work on trunk.

The problem I see here is that code & branches being worked on for a year are 
now (apparently) close to being done and we are telling them to hold for 7 more 
months - this is not a reasonable ask..

If you are advocating for a 3.1 plan, I’m sure one of these branch ‘owners’ can 
volunteer. But this is how you get competing releases and split bandwidth.

As for compatibility / testing etc, it seems like there is a belief that the 
current ‘scoped’ features are all tested well in these areas and so adding more 
is going to hurt the release. There is no way this is the reality, trunk has so 
many features that have been landing for years, the only way we can 
collectively attempt towards making this stable is by getting as many parties 
together as possible, each verifying stuff that they need. Not by excluding 
specific features.

+Vinod

> This is one of those curse-of-cadence things: The higher your release 
> cadence, the less pressure to get "everything in". With a slower cadence, 
> more pressure to get stuff in, more pressure to hold up the release, slows 
> the cadence, gets even more stuff in, etc. etc.
> 
> - Andrew has been working on the release for months, we all need to 
> appreciate how much hard work that is and has been, especially for what is 
> going to be a major release.
> 
> - We know that things will be unstable in 3.0; Andrew's concern is about 
> making sure that the newest, unstablest (?) features can at least be bypassed 
> if there are problems. I we should also call out in the release notes what we 
> think are the unstable bits where people need to use caution (example: 
> S3Guard in "authoritative" mode)
> 
> - Anything related to wire compatibility has been problematic in the past; I 
> think it's essential that whatever packets get sent around are going to be 
> stable, so changes there need to be in, or at least the payloads set up ready 
> for the features. Same for new public APIs.
> 
> - As fpr the rest, I don't know. I think being strict about it and ruthless 
> in assessing the feature's stability & consequences of postponing the feature 
> until a Hadoop 3.1 release in Jan/Feb, with a plan to ship then and follow up 
> with a 3.2 in the summer.
> 
> Then: start planning that 3.1 release. Maybe I should put my hand up as 
> release manager for that one. Then everyone would realise how amenable Andrew 
> is being today.
> 
> 
> One other thing: alongside the big branches, there's the eternal backlog of 
> small patches. We should organise spending a few days updating, reviewing & 
> merging them in
> 
> -Steve
> 
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org 
> 
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org 
> 

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Andrew,

Each of the branches below have been created more than a year ago (!) and have 
been consistently worked upon and are now finally seeing the light of the day. 
When they are "few weeks” away, pushing them out by 7 *more* months just 
doesn’t make sense.

While I deeply appreciate the push for the dates, we shouldn’t be putting 
moratorium on features like this till the proposed date is due. As a release 
manager, it’s good to say that we will release a specific version as soon as 
so-and-so features are in, but let’s not put exclusions like this.

I propose that, as we have done with every release so far, (and more 
specifically the way we did 2.x alphas, betas, GA back in the day), we float 
the date, let the individual branch contributors decide their fate. As long as 
of course, they meet the timelines and the usual branch merge bar, testing / 
compatibility / impact on rest of the code-base etc.

Anything short of that, we will just be incentivizing contributors away from 
doing work in branches and towards putting stuff directly into trunk.

+Vinod

> On Aug 18, 2017, at 6:22 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> As you might have seen, we've had a number of branch merges floated this
> past week targeted for 3.0.0-beta1, which is planned for about a month from
> now.
> 
> In total, I'm currently tracking these branches:
> 
> YARN-2915: YARN federation (recently merged)
> HADOOP-13345: S3Guard (currently being voted on)
> YARN-5355: TSv2 alpha2 ("few weeks")
> YARN-5079: Native services ("few weeks")
> YARN-3926: Resource profiles ("few weeks")
> 
> We should effectively be in code freeze (only blockers/criticals), so the
> volume of merge proposals at this point came as a surprise. Despite our
> best efforts as software engineers, big code movement always comes with
> risk.
> 
> Since we started the 3.0 release series close to a year ago, I'm also loath
> to increase the scope. The main motivation for 3.0 was to deliver JDK8 and
> HDFS EC, and our users deserve a production-quality release with these
> features. We've also been good about the release cadence thus far in 3.x,
> so a 3.1 isn't that far out.
> 
> Here's my proposal:
> 
> * 3.0.0-beta1 includes S3Guard and YARN federation. Target date remains
> mid-Sept.
> * 3.0.0-GA includes TSv2 alpha2. Target date remains early November.
> * Everything else waits for 3.1, approximately March 2018.
> 
> My rationale for inclusion and exclusion is as follows:
> 
> Inclusion:
> 
> * YARN federation has been run in production, does not touch existing code,
> adds no new APIs, and is off by default.
> * S3Guard has been run in production and is off by default.
> * The first iteration of TSv2 was shipped in 3.0.0-alpha1, so we're
> committed to this for 3.0.0 GA. It's off by default and adds no impact.
> 
> Exclusion:
> 
> * The primary reason for exclusion is to maintain the planned release
> schedule. Native services and resource profiles are still a few weeks from
> being ready for merge.
> * A reminder that 3.1 is only another 3 months after 3.0 given our release
> cadence thus far. If there's demand, we could even do a 3.1 immediately
> following 3.0.
> 
> I'm happy to talk with the contributors of each of these features to
> understand their timelines and requirements, with the caveat that I'll be
> out through Wednesday next week. Please reach out.
> 
> Best,
> Andrew


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager

[Vinod Kumar Vavilapalli (JIRA)]

Vinod Kumar Vavilapalli created YARN-6930:
-

 Summary: Admins should be able to explicitly enable specific 
LinuxContainerRuntime in the NodeManager
 Key: YARN-6930
 URL: https://issues.apache.org/jira/browse/YARN-6930
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli


Today, in the java land, all LinuxContainerRuntimes are always enabled when 
using LinuxContainerExecutor and the user can simply invoke anything that 
he/she wants - default, docker, java-sandbox.

We should have a way for admins to explicitly enable only specific runtimes 
that he/she decides for the cluster. And by default, we should have everything 
other than the default one disabled.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Vinod Kumar Vavilapalli]

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du <j...@hortonworks.com> wrote:
> 
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
> 
> Thanks,
> 
> Junping
> 
> From: Jason Lowe <jl...@yahoo-inc.com.INVALID>
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
> yarn-dev@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
> 
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
> 
> Jason
> 
> 
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
> <kih...@yahoo-inc.com.INVALID> wrote:
> 
> 
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
> 
> On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du <j...@hortonworks.com> 
> wrote:
> 
> Hi all,
>Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
> released today which is a special security release. Now, we should work 
> towards 2.8.2 release which aim for production deployment. The focus 
> obviously is to fix blocker/critical issues [2], bug-fixes and *no* features 
> / improvements. We currently have 13 blocker/critical issues, and 10 of them 
> are Patch Available.
> 
>  I plan to cut an RC in a month - target for releasing before end of Aug., to 
> give enough time for outstanding blocker / critical issues. Will start moving 
> out any tickets that are not blockers and/or won't fit the timeline. For 
> progress of releasing effort, please refer our release wiki [2].
> 
>  Please share thoughts if you have any. Thanks!
> 
> Thanks,
> 
> Junping
> 
> [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
> [2] 2.8 Release wiki: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> 
> 
> From: Vinod Kumar Vavilapalli <vino...@apache.org>
> Sent: Thursday, July 20, 2017 1:05 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released
> 
> Hi all,
> 
> The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
> page: http://hadoop.apache.org/releases.html#Download
> This is a security release in the 2.8.0 release line. It consists of 2.8.0 
> plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.
> 
> Please note that 2.8.x release line continues to be not yet ready for 
> production use. Critical issues are being ironed out via testing and 
> downstream adoption. Production users should wait for a subsequent release in 
> the 2.8.x line.
> 
> Thanks
> +Vinod
> 
> 
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: About 2.7.4 Release

[Vinod Kumar Vavilapalli]

Thanks for taking 2.7.4 over Konstantin!

Regarding rolling RC next week, I still see that there are 4 blocker / critical 
tickets targeted for 2.7.4: 
https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%202.7.4
 
.

We should get closure on them. https://issues.apache.org/jira/browse/HDFS-11742 
 definitely was something 
that was deemed a blocker for 2.8.2, not sure about 2.7.4.

I’m ‘back’ - let me know if you need any help.

Thanks
+Vinod

> On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko  wrote:
> 
> Hi everybody.
> 
> We have been doing some internal testing of Hadoop 2.7.4. The testing is
> going well.
> Did not find any major issues on our workloads.
> Used an internal tool called Dynamometer to check NameNode performance on
> real cluster traces. Good.
> Overall test cluster performance looks good.
> Some more testing is still going on.
> 
> I plan to build an RC next week. If there are no objection.
> 
> Thanks,
> --Konst
> 
> On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko 
> wrote:
> 
>> Hey guys.
>> 
>> An update on 2.7.4 progress.
>> We are down to 4 blockers. There is some work remaining on those.
>> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
>> Would be good if people could follow up on review comments.
>> 
>> I looked through nightly Jenkins build results for 2.7.4 both on Apache
>> Jenkins and internal.
>> Some test fail intermittently, but there no consistent failures. I filed
>> HDFS-11985 to track some of them.
>> https://issues.apache.org/jira/browse/HDFS-11985
>> I do not currently consider these failures as blockers. LMK if some of
>> them are.
>> 
>> We started internal testing of branch-2.7 on one of our smallish (100+
>> nodes) test clusters.
>> Will update on the results.
>> 
>> There is a plan to enable BigTop for 2.7.4 testing.
>> 
>> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
>> Thank you everybody for contributing to this effort.
>> 
>> Regards,
>> --Konstantin
>> 
>> 
>> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka 
>> wrote:
>> 
>>> Sure.
>>> If you want to edit the wiki, please tell me your ASF confluence account.
>>> 
>>> -Akira
>>> 
>>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>>> 
 Couple of more JIRAs need to be back ported for 2.7.4 release. These will
 solve RM HA unstability issues.
 https://issues.apache.org/jira/browse/YARN-5333
 https://issues.apache.org/jira/browse/YARN-5988
 https://issues.apache.org/jira/browse/YARN-6304
 
 I will raise a JIRAs to back port it.
 
 @Akira , could  you help to add these JIRAs into wiki?
 
 Thanks & Regards
 Rohith Sharma K S
 
 On 29 May 2017 at 12:19, Akira Ajisaka  wrote:
 
 Created a page for 2.7.4 release.
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
> 
> If you want to edit this wiki, please ping me.
> 
> Regards,
> Akira
> 
> 
> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
> 
> Hi Konstantin Shvachko
>> 
>> 
>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>> trunk in following link.??
>> 
>> 
>> https://cwiki.apache.org/confluence/display/HADOOP
>> 
>> 
>> 
>> From: Konstantin Shvachko 
>> Sent: Saturday, May 13, 2017 3:58 AM
>> To: Akira Ajisaka
>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
>> yarn-dev@hadoop.apache.org
>> Subject: Re: About 2.7.4 Release
>> 
>> Latest update on the links and filters. Here is the correct link for
>> the
>> filter:
>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>> requestId=12340814
>> 
>> Also updated: https://s.apache.org/Dzg4
>> 
>> Had to do some Jira debugging. Sorry for confusion.
>> 
>> Thanks,
>> --Konstantin
>> 
>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>> shv.had...@gmail.com>
>> wrote:
>> 
>> Hey Akira,
>> 
>>> 
>>> I didn't have private filters. Most probably Jira caches something.
>>> Your filter is in the right direction, but for some reason it lists
>>> only
>>> 22 issues, while mine has 29.
>>> It misses e.g. YARN-5543 >> a/browse/YARN-5543>
>>> .
>>> 
>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release 

[jira] [Resolved] (YARN-6543) yarn application's privilege is determined by yarn process creator instead of yarn application user.

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-6543.
---
Resolution: Not A Bug

Agree with [~rohithsharma], this is working as designed. Closing this.

> yarn application's privilege is determined by yarn process creator instead of 
> yarn application user.
> 
>
> Key: YARN-6543
> URL: https://issues.apache.org/jira/browse/YARN-6543
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: wuchang
>
> My application is a pyspark application which is impersonated by user 
> 'wuchang'
> My application infomation is :
> {code}
> Application Report : 
> Application-Id : application_1493004858240_0007
> Application-Name : livy-session-6
> Application-Type : SPARK
> User : wuchang
> Queue : root.wuchang
> Start-Time : 1493708942748
> Finish-Time : 0
> Progress : 10%
> State : RUNNING
> Final-State : UNDEFINED
> Tracking-URL : http://10.120.241.82:34462
> RPC Port : 0
> AM Host : 10.120.241.82
> Aggregate Resource Allocation : 4369480 MB-seconds, 2131 vcore-seconds
> Diagnostics :
> {code}
> And the process is :
> {code}
> appuser  25454 25872  0 15:09 ?00:00:00 bash 
> /data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_04/default_container_executor.sh
> appuser  25456 25454  0 15:09 ?00:00:00 /bin/bash -c 
> /home/jdk/bin/java -server -Xmx1024m 
> -Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_04/tmp
>  '-Dspark.ui.port=0' '-Dspark.driver.port=40969' 
> -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_04
>  -XX:OnOutOfMemoryError='kill %p' 
> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url 
> spark://CoarseGrainedScheduler@10.120.241.82:40969 --executor-id 2 --hostname 
> 10.120.241.18 --cores 1 --app-id application_1493004858240_0007 
> --user-class-path 
> file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_04/__app__.jar
>  --user-class-path 
> file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_04/livy-api-0.3.0-SNAPSHOT.jar
>  --user-class-path 
> file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_04/livy-rsc-0.3.0-SNAPSHOT.jar
>  --user-class-path 
> file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_04/netty-all-4.0.29.Final.jar
>  --user-class-path 
> file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_04/commons-codec-1.9.jar
>  --user-class-path 
> file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_04/livy-core_2.11-0.3.0-SNAPSHOT.jar
>  --user-class-path 
> file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_04/livy-repl_2.11-0.3.0-SNAPSHOT.jar
>  1> 
> /home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_04/stdout
>  2> 
> /home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_04/stderr
> appuser  25468 25456  2 15:09 ?00:00:09 /home/jdk/bin/java -server 
> -Xmx1024m 
> -Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_04/tmp
>  -Dspark.ui.port=0 -Dspark.driver.port=40969 
> -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_04
>  -XX:OnOutOfMemoryError=kill %p 
> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url 
> spark://CoarseGrainedScheduler@10.120.241.82:40969 --executor-id 2 --hostname 
> 10.120.241.18 --cores 1 --app-id application_1493004858240_0007 
> --user-class-path 
> file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC2)

[Vinod Kumar Vavilapalli]

Thanks for taking the mantle from me on 2.8.0 and some persistent work getting 
2.8.0 out the door, Junping!

Apologies for bringing this up late, but I’d like to add one comment.

We should repeat what we did for 2.7.0 and In line with our experience there, 
we should annotate this release as not ready for production use. See the 
releases page - 
http://hadoop.apache.org/releases.html#25+August%2C+2016%3A+Release+2.7.3+available
 for our messaging on 2.7.0.

The expectation is that more downstream projects pick up the bits, iron out any 
incompatibilities we might have missed, and production users then pick up a 
solid 2.8.1.

Thanks
+Vinod

> On Mar 14, 2017, at 1:41 AM, Junping Du  wrote:
> 
> Hi all,
> With several important fixes get merged last week, I've created a new 
> release candidate (RC2) for Apache Hadoop 2.8.0.
> 
> This is the next minor release to follow up 2.7.0 which has been released 
> for more than 1 year. It comprises 2,919 fixes, improvements, and new 
> features. Most of these commits are released for the first time in branch-2.
> 
>  More information about the 2.8.0 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> 
>  Please note that RC0 and RC1 are not voted public because significant 
> issues are found just after RC tag getting published.
> 
>  The RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.0-RC2
> 
>  The RC tag in git is: release-2.8.0-RC2
> 
>  The maven artifacts are available via repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1056
> 
>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 03/20/2017 PDT time.
> 
> Thanks,
> 
> Junping


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Queries on YARN

[Vinod Kumar Vavilapalli]

Hi Sabiya,

> 1. Can we define our custom logical resources in YARN other than Memory &
> cpu core?  Is this flexibility there?

There is an ongoing effort to satisfy this very requirement - you can follow 
https://issues.apache.org/jira/browse/YARN-3926 
.

> 2. What is a role of HDFS in yarn, Can yarn work without HDFS?

YARN depends on a FileSystem (of which HDFS is an implementation) for two things
 (a) to find out where the application dependent artifacts (like jars, 
configurations, binaries etc) are present and downloadable to individual nodes 
from
 (b) to determine a location where it can upload application logs once an 
application finishes.

HTH
+Vinod

Re: About 2.7.4 Release

[Vinod Kumar Vavilapalli]

I was planning to take this up, celebrating my return from my paternity leave 
of absence for quite a while.

Marton, let me know if you do want to take this up instead and we can work 
together.

Thanks
+Vinod

> On Mar 7, 2017, at 9:13 AM, Sangjin Lee  wrote:
> 
> If we have a volunteer for releasing 2.7.4, we should go full speed
> ahead. We still need a volunteer from a PMC member or a committer as some
> tasks may require certain privileges, but I don't think it precludes
> working with others to close down the release.

[jira] [Resolved] (YARN-5068) Expose scheduler queue to application master

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-5068.
---
Resolution: Duplicate

Closing this accurately as a dup of YARN-1623.

> Expose scheduler queue to application master
> 
>
> Key: YARN-5068
> URL: https://issues.apache.org/jira/browse/YARN-5068
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6692.patch, YARN-5068.1.patch, 
> YARN-5068.2.patch, YARN-5068-branch-2.1.patch
>
>
> The AM needs to know the queue name in which it was launched.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-5759) Capability to register for a notification/callback on the expiry of timeouts

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-5759.
---
Resolution: Duplicate

Agree with [~rohithsharma]. Closing this as a dup of YARN-2261.

> Capability to register for a notification/callback on the expiry of timeouts
> 
>
> Key: YARN-5759
> URL: https://issues.apache.org/jira/browse/YARN-5759
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Gour Saha
>
> There is a need for the YARN native services REST-API service, to take 
> certain actions once a timeout of an application expires. For example, an 
> immediate requirement is to destroy a Slider application, once its lifetime 
> timeout expires and YARN has stopped the application. Destroying a Slider 
> application means cleanup of Slider HDFS state store and ZK paths for that 
> application. 
> Potentially, there will be advanced requirements from the REST-API service 
> and other services in the future, which will make this feature very handy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache YARN committers/contribu­t­ors meetup #4 (10/27)

[Vinod Kumar Vavilapalli]

Thanks to everyone who joined this meetup!

We had quite a blast both in the western hemisphere and from what I hear in the 
IST timezone too.

Overall, stats

 - PST
— Started at 269 patch-available tickets and got it down to 170 - a mix of 
commits, reviews + updates, closing invalid / not-applicable JIRAs
— Working notes: 
https://docs.google.com/spreadsheets/d/1kPKsm3VSnkLU107t-CL05RQ9xQxc6kN-h6o9bDrheaY/edit#gid=2076540402

 - IST: (Notes from Sunil)
— 19 patch commits, 11 added / rebased patches, and more commits on the way 
waiting for Jenkins
— Working notes: 
https://docs.google.com/spreadsheets/d/1EVga79x-sxrfxWoe3o_ZkyLgHbrzHJqAU4hq6qabW_k/edit?ts=5811eb51#gid=0

Special thanks
 - To Subru for sponsoring the event, logistics and a great lunch!
 - To Sunil for taking the initiative, organizing and running the contributors’ 
meetup in India!

We are thinking of doing this at a regular cadence but for a smaller duration 
than a full-day.

Thanks
+Vinod

> On Oct 19, 2016, at 11:08 AM, Subru Krishnan  wrote:
> 
> Folks,
> 
> Hope everyone's is doing great.
> 
> We are putting in one full day (5-6 hours) for a YARN review / commit
> marathon on *next Thursday, 27th Oct*.
> 
>Expected Audience: *regular contributor / committer in YARN*.
> 
>Non-audience: While the meetups are generally open to the general
> public, this is not a 'meetup to learn about YARN'.
> 
>Specific Agenda: YARN bug bash
> 
>Location: Microsoft Moffett Towers, 1020 Enterprise Way, Sunnyvale,
> CA.
> 
>Webex/Skype details for those who are remote: TBD
> 
>Meetup URL:
> http://www.meetup.com/Hadoop-Contributors/events/234971372/
> 
> IMPORTANT NOTES:
> - Food will be provided for the reviewers / committers to make sure they
> stay up :)
> - We have capacity for only *25 people*, so this isn't a walk-in, please *RSVP
> *and reach out to us if you want to join this meetup.
> 
> Thanks.


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-3755) Log the command of launching containers

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-3755.
---
Resolution: Duplicate

Actually I think this is a dup of YARN-4309 which now dumps a lot more useful 
information in addition to command line, env etc.

Closing this as duplicate for now, please reopen if you disagree.

> Log the command of launching containers
> ---
>
> Key: YARN-3755
> URL: https://issues.apache.org/jira/browse/YARN-3755
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>  Labels: oct16-easy
> Attachments: YARN-3755-1.patch, YARN-3755-2.patch
>
>
> In the resource manager log, yarn would log the command for launching AM, 
> this is very useful. But there's no such log in the NN log for launching 
> containers. It would be difficult to diagnose when containers fails to launch 
> due to some issue in the commands. Although user can look at the commands in 
> the container launch script file, this is an internal things of yarn, usually 
> user don't know that. In user's perspective, they only know what commands 
> they specify when building yarn application. 
> {code}
> 2015-06-01 16:06:42,245 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command 
> to launch container container_1433145984561_0001_01_01 : 
> $JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true 
> -Dhadoop.metrics.log.level=WARN  -Xmx1024m  
> -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator 
> -Dlog4j.configuration=tez-container-log4j.properties 
> -Dyarn.app.container.log.dir= -Dtez.root.logger=info,CLA 
> -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster 
> 1>/stdout 2>/stderr
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-2981) DockerContainerExecutor must support a Cluster-wide default Docker image

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-2981.
---
Resolution: Won't Fix

Agreed. I am closing this as "WON'T FIX" for now given YARN-5388. Please reopen 
this if you disagree. Thanks.

> DockerContainerExecutor must support a Cluster-wide default Docker image
> 
>
> Key: YARN-2981
> URL: https://issues.apache.org/jira/browse/YARN-2981
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Abin Shahab
>Assignee: Abin Shahab
>  Labels: oct16-easy
> Attachments: YARN-2981.patch, YARN-2981.patch, YARN-2981.patch, 
> YARN-2981.patch
>
>
> This allows the yarn administrator to add a cluster-wide default docker image 
> that will be used when there are no per-job override of docker images. With 
> this features, it would be convenient for newer applications like slider to 
> launch inside a cluster-default docker container.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Vinod Kumar Vavilapalli]

Here’s my +1 to conclude the vote.

With 8 binding +1s, 9 non-binding +1s and no -1s, this vote passes.

I’ll push the RC forward to an official release.

Thanks to everyone who voted!

+Vinod

> On Aug 24, 2016, at 10:30 PM, Wangda Tan <wheele...@gmail.com> wrote:
> 
> +1 Binding.
> 
> - Built and deploy a single node cluster from source code.
> - Ran some example jobs.
> 
> Thanks,
> Wangda
> 
> On Tue, Aug 23, 2016 at 12:35 PM, Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
> 
>> +1 (non-binding)
>> 
>> - Downloaded source and tar verified the checksum
>> - Installed pseudo cluster with Node Labels and ATSV1 enabled.
>> - Verified with running MR apps with and without labels.
>> - Verified ATSV1 UI.
>> 
>> Regards,
>> + Naga
>> 
>> On Mon, Aug 22, 2016 at 11:55 PM, Jason Lowe <jl...@yahoo-inc.com.invalid>
>> wrote:
>> 
>>> +1 (binding)
>>> - Verified signatures and digests- Successfully built from source with
>>> native support- Deployed a single-node cluster- Ran some sample jobs
>>> successfully
>>> 
>>> Jason
>>> 
>>>  From: Vinod Kumar Vavilapalli <vino...@apache.org>
>>> To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>;
>>> hdfs-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; "
>>> mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org>
>>> Cc: Vinod Kumar Vavilapalli <vino...@apache.org>
>>> Sent: Wednesday, August 17, 2016 9:05 PM
>>> Subject: [VOTE] Release Apache Hadoop 2.7.3 RC2
>>> 
>>> Hi all,
>>> 
>>> I've created a new release candidate RC2 for Apache Hadoop 2.7.3.
>>> 
>>> As discussed before, this is the next maintenance release to follow up
>>> 2.7.2.
>>> 
>>> The RC is available for validation at: http://home.apache.org/~
>>> vinodkv/hadoop-2.7.3-RC2/ <http://home.apache.org/~
>>> vinodkv/hadoop-2.7.3-RC2/>
>>> 
>>> The RC tag in git is: release-2.7.3-RC2
>>> 
>>> The maven artifacts are available via repository.apache.org <
>>> http://repository.apache.org/> at https://repository.apache.org/
>>> content/repositories/orgapachehadoop-1046 <https://repository.apache.
>>> org/content/repositories/orgapachehadoop-1046>
>>> 
>>> The release-notes are inside the tar-balls at location
>>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
>>> hosted this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/
>>> releasenotes.html <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/
>>> releasenotes.html> for your quick perusal.
>>> 
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - few more issues with RC1 forced a RC2 [2]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
>>> caused 2.7.3 (along with every other Hadoop release) to slip by quite a
>>> bit. This release's related discussion thread is linked below: [3].
>>> 
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>> 
>>> Thanks,
>>> Vinod
>>> 
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0:
>> https://www.mail-archive.com/
>>> hdfs-dev%40hadoop.apache.org/index.html#26106 <
>>> https://www.mail-archive.com/hdfs-dev@hadoop.apache.org/index.html#26106
>>> 
>>> [2] [VOTE] Release Apache Hadoop 2.7.3 RC1:
>> https://www.mail-archive.com/
>>> hdfs-dev%40hadoop.apache.org/msg26336.html <
>> https://www.mail-archive.com/
>>> hdfs-...@hadoop.apache.org/msg26336.html>
>>> [3] 2.7.3 release plan: https://www.mail-archive.com/
>>> hdfs-dev%40hadoop.apache.org/msg24439.html <http://markmail.org/thread/
>>> 6yv2fyrs4jlepmmr>
>>> 
>>> 
>>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-1763) Handle RM failovers during the submitApplication call.

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-1763.
---
Resolution: Duplicate

> Handle RM failovers during the submitApplication call.
> --
>
> Key: YARN-1763
> URL: https://issues.apache.org/jira/browse/YARN-1763
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-1836) Add retry cache support in ResourceManager

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved YARN-1836.
---
Resolution: Invalid

Old JIRA.

As [~xgong] [mentioned on 
YARN-1521|https://issues.apache.org/jira/browse/YARN-1521?focusedCommentId=13948528=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13948528],
 YARN doesn't need yet another RetryCache - StateStore + recovered apps already 
serve that purpose.

Closing this as invalid for now. We can reopen it in future as needed. Revert 
back if you disagree. Tx.

> Add retry cache support in ResourceManager
> --
>
> Key: YARN-1836
> URL: https://issues.apache.org/jira/browse/YARN-1836
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Tsuyoshi Ozawa
>Assignee: Tsuyoshi Ozawa
>
> HDFS-4942 supports RetryCache on NN. This JIRA tracks RetryCache on 
> ResourceManager. If the RPCs are non-idempotent, we should use RetryCache to 
> avoid returning incorrect failures to client.
> YARN-1521 is a related JIRA. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Vinod Kumar Vavilapalli]

Too late for 2.7.3 - I want us to quickly get back to a regular cadence of 
releases.

Doesn’t look like a regression IAC. Let’s put it in the next release if needed.

Thanks
+Vinod

> On Aug 22, 2016, at 6:19 AM, Brahma Reddy Battula 
>  wrote:
> 
> Felt like HDFS-8388, should be in for 2.7.3

[VOTE] Release Apache Hadoop 2.7.3 RC2

[Vinod Kumar Vavilapalli]

Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 


The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - few more issues with RC1 forced a RC2 [2]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 

[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Canceling the release vote for this and other issues reported.

+Vinod

> On Aug 16, 2016, at 10:01 PM, Akira Ajisaka <ajisa...@oss.nttdata.co.jp> 
> wrote:
> 
> -1 (binding)
> 
> HADOOP-13434 and HADOOP-11814, committed between RC0 and RC1, are not 
> reflected in the release note.
> 
> -Akira
> 
> On 8/17/16 13:29, Allen Wittenauer wrote:
>> 
>> 
>> -1
>> 
>> HDFS-9395 is an incompatible change:
>> 
>> a) Why is not marked as such in the changes file?
>> b) Why is an incompatible change in a micro release, much less a minor?
>> c) Where is the release note for this change?
>> 
>> 
>>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vino...@apache.org> 
>>> wrote:
>>> 
>>> Hi all,
>>> 
>>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>>> 
>>> As discussed before, this is the next maintenance release to follow up 
>>> 2.7.2.
>>> 
>>> The RC is available for validation at: 
>>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
>>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>> 
>>> The RC tag in git is: release-2.7.3-RC1
>>> 
>>> The maven artifacts are available via repository.apache.org 
>>> <http://repository.apache.org/> at 
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
>>> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>>> 
>>> The release-notes are inside the tar-balls at location 
>>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>>> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
>>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for 
>>> your quick perusal.
>>> 
>>> As you may have noted,
>>> - few issues with RC0 forced a RC1 [1]
>>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
>>> caused 2.7.3 (along with every other Hadoop release) to slip by quite a 
>>> bit. This release's related discussion thread is linked below: [2].
>>> 
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>> 
>>> Thanks,
>>> Vinod
>>> 
>>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
>>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
>>> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>>> [2]: 2.7.3 release plan: 
>>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>>> <http://markmail.org/thread/6yv2fyrs4jlepmmr>
>> 
>> 
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

I always look at CHANGES.txt entries for incompatible-changes and this JIRA 
obviously wasn’t there.

Anyways, this shouldn’t be in any of branch-2.* as committers there clearly 
mentioned that this is an incompatible change.

I am reverting the patch from branch-2* .

Thanks
+Vinod

> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <a...@effectivemachines.com> 
> wrote:
> 
> 
> 
> -1
> 
> HDFS-9395 is an incompatible change:
> 
> a) Why is not marked as such in the changes file?
> b) Why is an incompatible change in a micro release, much less a minor?
> c) Where is the release note for this change?
> 
> 
>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vino...@apache.org> 
>> wrote:
>> 
>> Hi all,
>> 
>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
>> 
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>> 
>> The RC is available for validation at: 
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>> 
>> The RC tag in git is: release-2.7.3-RC1
>> 
>> The maven artifacts are available via repository.apache.org 
>> <http://repository.apache.org/> at 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
>> 
>> The release-notes are inside the tar-balls at location 
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>> hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for 
>> your quick perusal.
>> 
>> As you may have noted,
>> - few issues with RC0 forced a RC1 [1]
>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
>> caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. 
>> This release's related discussion thread is linked below: [2].
>> 
>> Please try the release and vote; the vote will run for the usual 5 days.
>> 
>> Thanks,
>> Vinod
>> 
>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
>> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>
>> [2]: 2.7.3 release plan: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>> <http://markmail.org/thread/6yv2fyrs4jlepmmr>
> 
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Thanks Steve, this is one area that isn’t very well release-tested usually!

+Vinod

> On Aug 16, 2016, at 2:25 AM, Steve Loughran  wrote:
> 
> I've just looked at the staged JARs and how they worked with downstream apps 
> —that being a key way that Hadoop artifacts are adopted.

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Thanks Marco. It was a Thursday late-night slip-up.

Fixed the dates and replaced the bits, so the voting can continue.

FYI, they aren’t binding though - as it all depends on how the release voting 
goes. One should usually only trust the release-date published on the website.

Thanks
+Vinod

> On Aug 13, 2016, at 1:35 PM, Marco Zühlke <mzueh...@gmail.com 
> <mailto:mzueh...@gmail.com>> wrote:
> 
> Hi Vinod,
> 
> I'm not sure if this is relevant, but you changed the release date in the 
> CHANGES.txt 
> <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f#diff-4de1a6452466a82b89570bd9ab606c12>
>  files to 2016-09-19.
> I guess you have meant 2016-08-19.
> 
> See: 
> https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f
>  
> <https://github.com/apache/hadoop/commit/5474c9e736d4c44a603a3f6749130b67cd4da52f>
> 
> 
> Thanks,
> Marco
> 
> 
> 
> 2016-08-12 18:45 GMT+02:00 Vinod Kumar Vavilapalli <vino...@apache.org 
> <mailto:vino...@apache.org>>:
> Hi all,
> 
> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.2.
> 
> The RC is available for validation at: 
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/> 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
> 
> The RC tag in git is: release-2.7.3-RC1
> 
> The maven artifacts are available via repository.apache.org 
> <http://repository.apache.org/> <http://repository.apache.org/ 
> <http://repository.apache.org/>> at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/> 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>>
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html> 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for 
> your quick perusal.
> 
> As you may have noted,
>  - few issues with RC0 forced a RC1 [1]
>  - a very long fix-cycle for the License & Notice issues (HADOOP-12893) 
> caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. 
> This release's related discussion thread is linked below: [2].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106> 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106>>
> [2]: 2.7.3 release plan: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
> 

[VOTE] Release Apache Hadoop 2.7.3 RC1

[Vinod Kumar Vavilapalli]

Hi all,

I've created a release candidate RC1 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 


The RC tag in git is: release-2.7.3-RC1

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2]: 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Vinod Kumar Vavilapalli]

But, everyone please do continue your sanity checking on RC0 in case there are 
more issues to be fixed.

Thanks
+Vinod

> On Jul 26, 2016, at 12:11 PM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
> Thanks Daniel and Wei.
> 
> I think these are worth fixing, I’m withdrawing this RC. Will look at fixing 
> these issues and roll a new candidate with the fixes as soon as possible.
> 
> Thanks
> +Vinod
> 
>> On Jul 26, 2016, at 11:05 AM, Wei-Chiu Chuang <weic...@apache.org 
>> <mailto:weic...@apache.org>> wrote:
>> 
>> I noticed two issues:
>> 
>> (1) I ran hadoop checknative, but it seems the binary tarball was not 
>> compiled with native library for Linux. On the contrary, the Hadoop built 
>> from source tarball with maven -Pnative can find the native libraries on the 
>> same host.
>> 
>> (2) I noticed that the release dates in CHANGES.txt in tag release-2.7.3-RC0 
>> are set to Release 2.7.3 - 2016-07-27.
>> However, the release dates in CHANGES.txt in the source and binary tar balls 
>> are set to Release 2.7.3 - 2016-08-01. This is probably a non-issue though.
>> 
>> * Downloaded source and binary.
>> * Verified signature.
>> * Verified checksum.
>> * Built from source using 64-bit Java 7 (1.7.0.75) and 8 (1.8.0.05). Both 
>> went fine.
>> * Ran hadoop checknative
>> 
>> On Tue, Jul 26, 2016 at 9:12 AM, Rushabh Shah 
>> <rusha...@yahoo-inc.com.invalid <mailto:rusha...@yahoo-inc.com.invalid>> 
>> wrote:
>> Thanks Vinod for all the release work !
>> +1 (non-binding).
>> * Downloaded from source and built it.* Deployed a pseudo distributed 
>> cluster.
>> * Ran some sample jobs: sleep, pi* Ran some dfs commands.* Everything works 
>> fine.
>> 
>> 
>> On Friday, July 22, 2016 9:16 PM, Vinod Kumar Vavilapalli 
>> <vino...@apache.org <mailto:vino...@apache.org>> wrote:
>> 
>> 
>>  Hi all,
>> 
>> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>> 
>> As discussed before, this is the next maintenance release to follow up 2.7.2.
>> 
>> The RC is available for validation at: 
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/> 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
>> 
>> The RC tag in git is: release-2.7.3-RC0
>> 
>> The maven artifacts are available via repository.apache.org 
>> <http://repository.apache.org/> <http://repository.apache.org/ 
>> <http://repository.apache.org/>> at 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/> 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
>> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/>>
>> 
>> The release-notes are inside the tar-balls at location 
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I 
>> hosted this at 
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
>> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html> 
>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
>> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for 
>> your quick perusal.
>> 
>> As you may have noted, a very long fix-cycle for the License & Notice issues 
>> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip 
>> by quite a bit. This release's related discussion thread is linked below: 
>> [1].
>> 
>> Please try the release and vote; the vote will run for the usual 5 days.
>> 
>> Thanks,
>> Vinod
>> 
>> [1]: 2.7.3 release plan: 
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
>> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> 
>> <http://markmail.org/thread/6yv2fyrs4jlepmmr 
>> <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
>> 
>>
>> 
> 

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Vinod Kumar Vavilapalli]

Thanks Daniel and Wei.

I think these are worth fixing, I’m withdrawing this RC. Will look at fixing 
these issues and roll a new candidate with the fixes as soon as possible.

Thanks
+Vinod

> On Jul 26, 2016, at 11:05 AM, Wei-Chiu Chuang <weic...@apache.org> wrote:
> 
> I noticed two issues:
> 
> (1) I ran hadoop checknative, but it seems the binary tarball was not 
> compiled with native library for Linux. On the contrary, the Hadoop built 
> from source tarball with maven -Pnative can find the native libraries on the 
> same host.
> 
> (2) I noticed that the release dates in CHANGES.txt in tag release-2.7.3-RC0 
> are set to Release 2.7.3 - 2016-07-27.
> However, the release dates in CHANGES.txt in the source and binary tar balls 
> are set to Release 2.7.3 - 2016-08-01. This is probably a non-issue though.
> 
> * Downloaded source and binary.
> * Verified signature.
> * Verified checksum.
> * Built from source using 64-bit Java 7 (1.7.0.75) and 8 (1.8.0.05). Both 
> went fine.
> * Ran hadoop checknative
> 
> On Tue, Jul 26, 2016 at 9:12 AM, Rushabh Shah <rusha...@yahoo-inc.com.invalid 
> <mailto:rusha...@yahoo-inc.com.invalid>> wrote:
> Thanks Vinod for all the release work !
> +1 (non-binding).
> * Downloaded from source and built it.* Deployed a pseudo distributed cluster.
> * Ran some sample jobs: sleep, pi* Ran some dfs commands.* Everything works 
> fine.
> 
> 
> On Friday, July 22, 2016 9:16 PM, Vinod Kumar Vavilapalli 
> <vino...@apache.org <mailto:vino...@apache.org>> wrote:
> 
> 
>  Hi all,
> 
> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> 
> As discussed before, this is the next maintenance release to follow up 2.7.2.
> 
> The RC is available for validation at: 
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/> 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>>
> 
> The RC tag in git is: release-2.7.3-RC0
> 
> The maven artifacts are available via repository.apache.org 
> <http://repository.apache.org/> <http://repository.apache.org/ 
> <http://repository.apache.org/>> at 
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/> 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 
> <https://repository.apache.org/content/repositories/orgapachehadoop-1040/>>
> 
> The release-notes are inside the tar-balls at location 
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
> this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
> <http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html> 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
> <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>> for 
> your quick perusal.
> 
> As you may have noted, a very long fix-cycle for the License & Notice issues 
> (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip 
> by quite a bit. This release's related discussion thread is linked below: [1].
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1]: 2.7.3 release plan: 
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 
> <https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html> 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr 
> <http://markmail.org/thread/6yv2fyrs4jlepmmr>>
> 
>
> 

[DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Vinod Kumar Vavilapalli]

Forking the thread to make sure it attracts enough eye-balls. The earlier one 
was about 3.0.0 specifically and I don’t think enough people were watching that.

I’ll try to summarize a bit.

# Today’s state of release numbering and ordering:
So far, all the releases we have done, we have followed a simple timeline 
ordering. By this I mean that we always lined up releases one after another. 
Due to this, it was relatively simple to follow the causal ordering of 
releases, and we could answer ordering questions like “when was this patch 
first committed”.

The first time this started getting hairy was with parallel 2.6.x and 2.7.x 
releases. We managed the confusion by ordering them one after another: 2.7.1 -> 
2.6.2 -> 2.6.3 -> 2.7.2 -> 2.6.4 -> 2.7.3. This worked okay with two concurrent 
releases. 

Yes, by doing this, we could no longer answer questions by simply looking 
at the release numbers. But Sangjin / Junping / myself who did these 2.x 
releases ordered them one after another to avoid further confusion to 
developers and still retain the ability to just go back, look at the history 
and quickly answer the question of "what happened before and what happened 
after”. It wasn’t perfect, but we did what we did to keep it under control.

# What’s coming
Obviously, with 2.8.0 and 3.0.0-alpha1 release trains starting, this 
confusion goes to a whole new level. We’ll have 2.6.x, 2.7.x, 2.8.x, 3.0.0.x 
(and 2.9.x if we chose to make one) and following the ordering becomes a 
nightmare. The related problems that were discussed in the original thread was 
about how we mark fix-versions for patches, and how we generate change-logs - 
the answers for both of these follow the solution to the root problem of 
concurrent releases.

# Proposals?
There were some discussions about semantic versioning in the thread form 
which I forked this one from, but it’s not clear to me how semantic versioning 
supports concurrent release trains.

IAC, it’s time we look at this afresh and if need be, make some new rules 
before we make more of these releases and make it that much harder to follow 
for everyone.

Thanks
+Vinod
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

+1

Thanks
+Vinod

> On Jul 26, 2016, at 7:39 AM, Wangda Tan  wrote:
> 
> lets try to use both jdiff and the new tool and compare results because this 
> is the first time with the new tool.
> 
> Appreciate your time to help us about this effort!

Re: [Release thread] 2.8.0 release activities

[Vinod Kumar Vavilapalli]

Thanks for all that great help moving this release forward, Jian, Sangjin, 
Wangda et al! Appreciate it.

The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0 last week.

I only see 9 blocker / critical fixes pending for 2.8.0 - 
https://issues.apache.org/jira/issues/?filter=12334985. Let’s do this!

Thanks
+Vinod

> On May 11, 2016, at 6:04 PM, Jian He <j...@hortonworks.com> wrote:
> 
> For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs attention 
> for 2.8
> 
> MAPREDUCE-6288
> YARN-1815
> YARN-4685
> YARN-4844
> 
> The rest are either improvements or long-standing issues and does not qualify 
> release blocker, IMO.
> I think we’ll try to get these 4 jiras in asap. The rest will be on best 
> effort, resolve as much as possible and move them out if not resolved in time.
> 
> Jian
> 
> On May 11, 2016, at 5:37 PM, Wangda Tan 
> <wheele...@gmail.com<mailto:wheele...@gmail.com>> wrote:
> 
> Sounds good to me :).
> 
> Jian and I have looked at all existing 2.8.0 blockers and criticals today.
> To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
> out. Left comments on these JIRAs asked original owners, plan to update
> target version of these JIRAs early next week.
> 
> Will keep this thread updated.
> 
> Thanks,
> Wangda
> 
> 
> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org>> wrote:
> 
> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
> report the list here. Then folks can chime in if you agree
> 
> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
> 
> Thanks,
> Sangjin
> 
> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan 
> <wheele...@gmail.com<mailto:wheele...@gmail.com>> wrote:
> 
> +1, we should close such staled JIRAs to avoid doing unnecessary checks
> for
> every releases.
> 
> I'm working on reviewing YARN/MR critical/blocker patches currently, it
> gonna very helpful if someone else can help with reviewing Common/HDFS
> JIRAs.
> 
> Thanks,
> Wangda
> 
> 
> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee 
> <sj...@apache.org<mailto:sj...@apache.org>> wrote:
> 
> Where do we stand in terms of closing out blocker/critical issues for
> 2.8.0? I still see 50 open JIRAs in Vinod's list:
> https://issues.apache.org/jira/issues/?filter=12334985
> 
> But I see a lot of JIRAs with no patches or very stale patches. It
> would be
> a good exercise to come up with the list of JIRAs that we need to block
> 2.8.0 for and focus our attention on closing them out. Thoughts?
> 
> Thanks,
> Sangjin
> 
> On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 
> <ste...@hortonworks.com<mailto:ste...@hortonworks.com>
> 
> wrote:
> 
> 
> On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
> vino...@apache.org<mailto:vino...@apache.org>>
> wrote:
> 
> We are not converging - there’s still 58 more. I need help from the
> community in addressing / review 2.8.0 blockers. If folks can start
> with
> reviewing Patch available tickets, that’ll be great.
> 
> 
> 
> 
> I'm still doing the s3a stuff, other people testing and reviewing this
> stuff welcome.
> 
> in particular, I could do with others playing with this patch of mine,
> which adds counters and things into S3a, based on the azure
> instrumentation
> 
> https://issues.apache.org/jira/browse/HADOOP-13028
> 
> 
> 
> 
> 
> 
> 
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

Please don’t do this for 2.8.0 against 2.7.0 till we change our existing 
release ordering conventions.

Till now, I made sure 2.8.0 only has issues that are not in 2.7.3 - per the 
2.8.0-follows-2.7.3 convention. As part of the release process, I will fix the 
remaining tickets to follow this same convention. Again till we create new 
rules.

Thanks
+Vinod

> On Jul 25, 2016, at 11:02 AM, Andrew Wang  wrote:
> 
> If I don't hear otherwise, I'd like to go ahead and do the bulk fix version
> update on JIRA for 3.0.0-alpha1, diffing from 2.7.0. Will wait until EOD
> tomorrow in case there's more discussion. If any one is willing to help
> review my script and JIRA queries, that'd also be great.
> 
> I can also do the 2.8.0 bulk update (diffing from 2.7.0), since it should
> be a very similar process.
> 
> Best,
> Andrew
> 
> 
> On Fri, Jul 22, 2016 at 9:50 PM, Allen Wittenauer 
> wrote:
> 
>> 
>>> On Jul 22, 2016, at 7:16 PM, Andrew Wang 
>> wrote:
>>> 
>>> Does this mean you find our current system of listing a JIRA as being
>> fixed in both a 2.6.x and 2.7.x to be confusing?
>> 
>>Nope.  I'm only confused when there isn't a .0 release in the fix
>> line.  When I see 2.6.x and 2.7.x I know that it was back ported to those
>> branches.  If I don't see a .0, I figure it's either a mistake or something
>> that was already fixed by another change in that major/minor branch.  It's
>> almost always the former, however.
>> 
>>> FWIW, my usecase is normally not "what is the earliest release that has
>> this fix?" but rather "is this fix in this release?". If it's easy to query
>> the latter, you can also determine the former. Some kind of query tool
>> could help here.
>> 
>>It literally becomes a grep if people commit the release data into
>> the source tree, the release data is correct, etc:
>> 
>> $ mvn install site  -Preleasedocs -Pdocs -DskipTests
>> $ grep issueid
>> hadoop-common-project/hadoop-common/src/site/markdown/release/*/CHANGES*
>> 
>>We should probably update the release process to make sure that
>> *in progress* release data is also committed when a .0 is cut.  That's
>> likely missing. Another choice would be to modify the pom to that runs
>> releasedocmaker to use a range rather than single version, but that gets a
>> bit tricky with release dates, how big of a range, etc.  Not impossible,
>> just tricky.  Probably needs to be script that gets run as part of
>> create-release, maybe?
>> 
>>(In reality, I do this grep against my git repo that generates the
>> change log data automatically.  This way it is always up-to-date and not
>> dependent upon release data being committed.  But that same grep could be
>> done with a JQL query just as easily.)
>> 
>>> For the release notes, am I correct in interpreting this as:
>>> 
>>> * diff a.0.0 from the previous x.y.0 release
>>> * diff a.b.0  from the previous a.0.0 or a.b.0 release
>>> * diff a.b.c from the previous a.b.0 or a.b.c release
>> 
>>Pretty much yes.
>> 
>>> Ray pointed me at the changelogs of a few other enterprise software
>> products, and this strategy seems pretty common. I like it.
>> 
>>It's extremely common, to the point that putting every fix for
>> every release touched is, at least to me, weird and extremely
>> unconventional.
>> 
>>> I realize now that this means a lot more JIRAs will need the 2.8.0 fix
>> version, since they only have 2.6.x and 2.7.x.
>> 
>>Yup.
>> 
>>>  This makes the fix rules actually pretty easy:  the lowest a.b.0
>> release and all non-.0 releases.
>>> 
>>> I think this needs to be amended to handle the case of multiple major
>> release branches, since we could have something committed for both 2.9.0
>> and 3.1.0. So "lowest a.b.0 release within each major version"?
>> 
>>Yeah, switching to effectively trunk-based development makes the
>> rules harder.  It's one of the reasons why the two big enterprisey
>> companies I worked at prior to working on Hadoop didn't really do
>> trunk-based for the vast majority of projects.  They always cut a branch
>> (or equivalent for that SCM) to delineate a break.   Given the amount of
>> ex-Sun folks involved in the early days of Hadoop, our pre-existing
>> development processes very much reflect that culture.
>> 
>>> This was true previously (no releases from trunk, trunk is versioned
>> a.0.0), but now that trunk is essentially a minor release branch, its fix
>> version needs to be treated as such.
>> 
>>Yeah, I misspoke a bit when dealing with a head-of-tree model.
>> 3.0.0-alpha1 will generate different notes than 3.0.0-alpha2, obviously.
>> Every 3.0.0-(label) release is effectively a major version in that case.
>> 
>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: 

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

>> There's a plan for more 3.0.0 alphas, so there's still time to help out
> before things settle down for beta. If 2.8.0 is ready to go, it should
> happen before even alpha2.

I wasn’t talk about my irresistible urge to help (which of course is there : )) 
, it was about making sure enough eye-balls look through this. If it absolutely 
cannot wait (which I don’t still understand why), we will just have to do 
double-shifts I guess.

>> 
>> Obviously, I am not making the case that this issue won’t happen ever. In
>> fact, this already happened with the parallel 2.6.x and 2.7.x releases. And
>> we precisely avoided major confusion there by lining up 2.7.2 behind 2.6.3
>> etc.
>> 
> 
> Could you clarify how lining up releases differently avoids the fix version
> issue?

It wouldn’t avoid the issue, it reduces it by quite a bit.

The versioning discussion happened a lot of times before and we never got to 
semantic versioning or any such convention. If it helps, we can revive the 
discussion. So far, all the releases have documented as the change-list only 
the issues “that are not in the last release, date-wise, whether in this major 
number or the ones below”. It is a simple timeline ordering, and it worked okay 
with two concurrent releases. So, if 2.8.0 happens before 3.0.0-alpha1, the 
additional changes that make the change-list for 3.0.0-alpha1 will be a delta 
of trunk-only changes.

Again, lining up releses doesn’t fix the core issue of running parallel (>2) 
releases - it just continues the current order of things. For now, the only 
tool we have is timeline ordering of 2 releases. The typical question from 
anyone is “which version did this get fixed in” and the answer is always found 
from JIRA + svn/git-log. And the fact that the (>2) concurrent releases is a 
new ground for this community (yes, you are hearing it right, this didn’t ever 
happen before for an extended period of time), we need to make some new rules 
before we make more of these releases and make it harder to follow for everyone.

Thanks
+Vinod
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

Actually, I wouldn’t trust this report as it stands today at all.

I quickly glanced the report, looking for what it highlights as incompatible. 
But the ones that I saw have private / visible for testing annotations. Other 
than acting as useless evidence for those lashing out on branch-2, this won’t 
do much good.

Whenever we start working towards switching to this tool, it should incorporate 
the same exclude-annotations logic that the jdiff code-path does today. Do you 
think that is possible?

Thanks
+Vinod

> On Jul 22, 2016, at 4:53 PM, Vinod Kumar Vavilapalli <vino...@apache.org> 
> wrote:
> 
>> Do you want this check from some particular branch-2 release? It
>> matters since the releases along branch-2 have themselves had some
>> noise[2].
>> 
>> [1]: https://github.com/lvc/japi-compliance-checker 
>> <https://github.com/lvc/japi-compliance-checker> 
>> <https://github.com/lvc/japi-compliance-checker 
>> <https://github.com/lvc/japi-compliance-checker>>
>> [2]: http://abi-laboratory.pro/java/tracker/timeline/hadoop/ 
>> <http://abi-laboratory.pro/java/tracker/timeline/hadoop/> 
>> <http://abi-laboratory.pro/java/tracker/timeline/hadoop/ 
>> <http://abi-laboratory.pro/java/tracker/timeline/hadoop/>>
>> 
>> -- 
>> busbey

[VOTE] Release Apache Hadoop 2.7.3 RC0

[Vinod Kumar Vavilapalli]

Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 


The RC tag in git is: release-2.7.3-RC0

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
 for your 
quick perusal.

As you may have noted, a very long fix-cycle for the License & Notice issues 
(HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by 
quite a bit. This release's related discussion thread is linked below: [1].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html