Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

[Gour Saha]

+1 for merge

-Gour


> On Sep 6, 2018, at 8:44 PM, Sunil G  wrote:
> 
> +1 for the merge.
> 
> - Sunil
> 
> 
> On Wed, Sep 5, 2018 at 6:01 PM Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
> 
>> Hi All,
>> Thanks for feedback folks, based on the positive response starting
>> a Vote thread for merging YARN-3409 to master.
>> 
>> Regards,
>> + Naga & Sunil
>> 
>>> On Wed, 5 Sep 2018 2:51 am Wangda Tan,  wrote:
>>> 
>>> +1 for the merge, it gonna be a great addition to 3.2.0 release. Thanks
>> to
>>> everybody for pushing this feature to complete.
>>> 
>>> Best,
>>> Wangda
>>> 
>>> On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt 
>>> wrote:
>>> 
 +1 for merge. Fetaure would be a good addition to 3.2 release.
 
 --
 Bibin A Chundatt
 M: +91-9742095715 <+91%2097420%2095715>> <+91%2097420%2095715>>
 E: bibin.chund...@huawei.com
 2012实验室-印研IT&Cloud BU分部
 2012 Laboratories-IT&Cloud BU Branch Dept.
 From:Naganarasimha Garla
 To:common-...@hadoop.apache.org,Hdfs-dev,yarn-...@hadoop.apache.org,
 mapreduce-dev@hadoop.apache.org,
 Date:2018-08-29 20:00:44
 Subject:[Discuss] Merge discussion for Node attribute support feature
 YARN-3409
 
 Hi All,
 
 We would like to hear your thoughts on merging “Node Attributes Support
>> in
 YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to
>> get
 it in for HADOOP 3.2.
 
 *Major work happened in this branch*
 
 YARN-6858. Attribute Manager to store and provide node attributes in RM
 YARN-7871. Support Node attributes reporting from NM to RM( distributed
 node attributes)
 YARN-7863. Modify placement constraints to support node attributes
 YARN-7875. Node Attribute store for storing and recovering attributes
 
 *Detailed Design:*
 
 Please refer [1] for detailed design document.
 
 *Testing Efforts:*
 
 We did detailed tests for the feature in the last few weeks.
 This feature will be enabled only when Node Attributes constraints are
 specified through SchedulingRequest from AM.
 Manager implementation will help to store and recover Node Attributes.
 This
 works with existing placement constraints.
 
 *Regarding to API stability:*
 
 All newly added @Public APIs are @Unstable.
 
 Documentation jira [3] could help to provide detailed configuration
 details. This feature works from end-to-end and we tested this in our
 local
 cluster. Branch code is run against trunk and tracked via [4].
 
 We would love to get your thoughts before opening a voting thread.
 
 Special thanks to a team of folks who worked hard and contributed
>> towards
 this efforts including design discussion / patch / reviews, etc.: Weiwei
 Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
 Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong
>> Chen.
 
 [1] :
 
 
>> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
 [2] : https://issues.apache.org/jira/browse/YARN-3409
 [3] : https://issues.apache.org/jira/browse/YARN-7865
 [4] : https://issues.apache.org/jira/browse/YARN-8718
 
 Thanks,
 + Naga & Sunil Govindan
 
>>> 
>> 

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Gour Saha]

Sorry for the really late send.

+1 (non-binding)

-Verified checksums and sha1
-Built from source and ran YARN Service core and api tests successfully
-Installed in a 7-node cluster
-Launched, flexed up/down, stopped, started and destroyed the sample sleeper 
service
-Launched, flexed, stopped, started Hive/LLAP
-Tested the Service REST APIs and command line

-Gour

On 4/5/18, 10:49 AM, "Wangda Tan"  wrote:

Thanks everybody for voting!
The vote passes successfully with 11 binding +1 votes, 8 non-binding +1
votes and no -1s.

I will work on the staging and releases.

Best,
Wangda Tan


On Thu, Apr 5, 2018 at 10:46 AM, Vinod Kumar Vavilapalli  wrote:

> That is a great observation. And I missed your previous email about the
> shaded vs unshaded jars already getting fixed.
>
> I guess we are good to go.
>
> 
> --
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
>
> Verification
> - [Check] Successful recompilation from source tar-ball
> - [Check] Signature verification
> -- Note: The format of the mds files changed a bit - not a biggie.
> -- For e.g, in 3.0.0 and 2.x releases, it has lines of the form "
> *hadoop-3.0.0-src.tar.gz*: SHA256 = 8B21AD79 50BD606B 2A7C91FB AE9FC279
> 7BCED50B B2600318 B7E0BE3A 74DFFF71"
> -- But in 3.1.0 RC it is, "
> */build/source/target/artifacts/hadoop-3.1.0.tar.gz*: SHA256 = 670D2CED
> 595FA42D 9FA1A93C 4E39B39F 47002CAD 1553D9DF 163EE828 CA5143E7"
> - [Check] Generating dist tarballs from source tar-ball
> - [Check] Testing
>-- Start NN, DN, RM, NM, JHS, Timeline Service
>-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
>-- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> +1 binding.
>
> Thanks
> +Vinod
>
> On Apr 3, 2018, at 8:13 PM, Wangda Tan  wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
>
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 289
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> /hadoop-yarn-api-.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan  wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal 
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> --
> From: Wangda Tan 
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal 
> Cc: Gera Shegalov , Sunil G , "

Re: [VOTE] Merge yarn-native-services branch into trunk

[Gour Saha]

+1 (non-binding)


-Gour

On 10/30/17, 1:49 PM, "Jian He"  wrote:

>Few more things:
>
>This is the document for trying a non-docker service on YARN.
>https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-pro
>ject/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/QuickStar
>t.md
>
>And the document for a docker based service
>https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-pro
>ject/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/Examples.
>md
>
>And the vote lasts 7 days as usual.
>
>Thanks,
>Jian
>
>On Oct 30, 2017, at 1:06 PM, Jian He
>mailto:j...@hortonworks.com>> wrote:
>
>Hi All,
>
>I would like to restart the vote for merging yarn-native-services to
>trunk.
>Since last vote, we have been working on several issues in documentation,
>DNS, CLI modifications etc. We believe now the feature is in a much
>better shape.
>
>Some back ground:
>At a high level, the following are the key feautres implemented.
>- YARN-5079[1]. A native YARN framework (ApplicationMaster) to
>orchestrate existing services to YARN either docker or non-docker based.
>- YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to
>deploy a service via a simple JSON spec
>- YARN-4757[3]. Extending today's service registry with a simple DNS
>service to enable users to discover services deployed on YARN via
>standard DNS lookup
>- YARN-6419[4]. UI support for native-services on the new YARN UI
>All these new services are optional and are sitting outside of the
>existing system, and have no impact on existing system if disabled.
>
>Special thanks to a team of folks who worked hard towards this: Billie
>Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
>Sharma K S, Sunil G, Akhil PB, Eric Yang. This effort could not be
>possible without their ideas and hard work.
>Also thanks Allen for some review and verifications.
>
>Thanks,
>Jian
>
>[1] https://issues.apache.org/jira/browse/YARN-5079
>[2] https://issues.apache.org/jira/browse/YARN-4793
>[3] https://issues.apache.org/jira/browse/YARN-4757
>[4] https://issues.apache.org/jira/browse/YARN-6419
>


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Merge yarn-native-services branch into trunk

[Gour Saha]

Thanks Allen. You are right, the github renderer does have trouble
rendering the headers. I was only looking at the html generated by mvn
site, which did not have trouble rendering them. Anyway I added a space
after all the hashes and it looks ok through github now.

-Gour 

On 9/5/17, 3:20 PM, "Allen Wittenauer"  wrote:

>
>> On Sep 5, 2017, at 3:12 PM, Gour Saha  wrote:
>> 
>> 2) Lots of markdown problems in the NativeServicesDiscovery.md document.
>> This includes things like Œyarnsite.xml¹ (missing a dash.)
>> 
>> The md patch uploaded to YARN-5244 had some special chars. I fixed those
>> in YARN-7161.
>
>
>   It’s a lot more than just special chars I think.  Even github (which has
>a way better markdown processor than what we’re using for the site docs)
>is having trouble rendering it:
>
>https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c64
>06ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/na
>tive-services/NativeServicesDiscovery.md
>
>e.g., all of those ‘###’ are likely missing a space.
>

Re: [VOTE] Merge yarn-native-services branch into trunk

[Gour Saha]

 2) Lots of markdown problems in the NativeServicesDiscovery.md document.
This includes things like Œyarnsite.xml¹ (missing a dash.)

The md patch uploaded to YARN-5244 had some special chars. I fixed those
in YARN-7161.


>


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Merge yarn-native-services branch into trunk

[Gour Saha]

+1 (non-binding)

On 9/1/17, 11:58 AM, "Billie Rinaldi"  wrote:

>+1 (non-binding)
>
>On Thu, Aug 31, 2017 at 8:33 PM, Jian He  wrote:
>
>> Hi All,
>>
>> I would like to call a vote for merging yarn-native-services to trunk.
>>The
>> vote will run for 7 days as usual.
>>
>> At a high level, the following are the key feautres implemented.
>> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate
>>and
>> orchestrate existing services to YARN either docker or non-docker based.
>> - YARN-4793[2]. A Rest API server for user to deploy a service via a
>> simple JSON spec
>> - YARN-4757[3]. Extending today's service registry with a simple DNS
>> service to enable users to discover services deployed on YARN
>> - YARN-6419[4]. UI support for native-services on the new YARN UI
>> All these new services are optional and are sitting outside of the
>> existing system, and have no impact on existing system if disabled.
>>
>> Special thanks to a team of folks who worked hard towards this: Billie
>> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
>>Sharma
>> K S, Sunil G, Akhil PB. This effort could not be possible without their
>> ideas and hard work.
>>
>> Thanks,
>> Jian
>>
>> [1] https://issues.apache.org/jira/browse/YARN-5079
>> [2] https://issues.apache.org/jira/browse/YARN-4793
>> [3] https://issues.apache.org/jira/browse/YARN-4757
>> [4] https://issues.apache.org/jira/browse/YARN-6419
>>
>>
>> -
>> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>>
>>


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org