[jira] [Created] (YARN-7491) Make sure AM is not scheduled on an opportunistic container

2017-11-13 Thread Haibo Chen (JIRA) 
Haibo Chen created YARN-7491:


 Summary: Make sure AM is not scheduled on an opportunistic 
container
 Key: YARN-7491
 URL: https://issues.apache.org/jira/browse/YARN-7491
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler
Reporter: Haibo Chen
Assignee: Haibo Chen






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7490) ConcurrentModificationException in AMRMClient

2017-11-13 Thread Jian He (JIRA) 
Jian He created YARN-7490:
-

 Summary: ConcurrentModificationException in AMRMClient
 Key: YARN-7490
 URL: https://issues.apache.org/jira/browse/YARN-7490
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He


It is observed that below exception occurred in service AM,  looks like the 
AMRMClientImpl -> ResourceRequestInfo -> containerRequests can be concurrently 
accessed without protection 

{code}
2017-11-13 19:48:10,487 [AMRM Callback Handler Thread] ERROR 
service.ServiceScheduler - Error in AMRMClient callback handler
java.util.ConcurrentModificationException
  at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:711)
  at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:734)
  at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
  at 
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.getMatchingRequests(AMRMClientImpl.java:680)
  at 
org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.getMatchingRequests(AMRMClientAsync.java:239)
  at 
org.apache.hadoop.yarn.service.ServiceScheduler$AMRMClientCallback.onContainersAllocated(ServiceScheduler.java:486)
  at 
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:359)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7489) ConcurrentModificationException in RMAppImpl#getRMAppMetrics

2017-11-13 Thread Tao Yang (JIRA) 
Tao Yang created YARN-7489:
--

 Summary: ConcurrentModificationException in 
RMAppImpl#getRMAppMetrics
 Key: YARN-7489
 URL: https://issues.apache.org/jira/browse/YARN-7489
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacityscheduler
Reporter: Tao Yang
Assignee: Tao Yang


The REST clients have sometimes failed to query applications through apps REST 
API in RMWebService and it happened when iterating 
attempts(RMWebServices#getApps --> AppInfo# --> 
RMAppImpl#getRMAppMetrics) and meanwhile these attempts 
changed(AttemptFailedTransition#transition --> 
RMAppImpl#createAndStartNewAttempt --> RMAppImpl#createNewAttempt). 
Application state changed within the lockup period of writeLock in RMAppImpl, 
so that we can add readLock before iterating attempts to fix this problem.
Error logs:
{noformat}
java.util.ConcurrentModificationException
at 
java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
at 
java.util.LinkedHashMap$LinkedValueIterator.next(LinkedHashMap.java:747)
at 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getRMAppMetrics(RMAppImpl.java:1487)
at 
org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo.(AppInfo.java:199)
at 
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getApps(RMWebServices.java:597)
at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at 
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at 
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at 
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at 
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at 
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Apache YARN committers/contribu­t­ors meetup #5

2017-11-13 Thread Karthik Kambatla 
Thanks for hosting this.

Should people respond to this email for date preferences or do you want to
use something like doodle?



On Sat, Nov 11, 2017 at 8:59 AM Daniel Templeton 
wrote:

> Sounds like there are no other volunteers, so we're happy to host here
> at Cloudera.  How about November 29th or December 6th in the afternoon?
> Suggested topics would include:
>
> 3.0 status/update/celebration
> 3.1 status, timing, and plans
> What we're doing with the 2.x branch going forward
> Docker status/plans
> Other?
>
> Thanks!
> Daniel
>
> On 10/20/17 1:54 PM, Daniel Templeton wrote:
> > Seems to me like we're due for a YARN contributors meetup.  Anyone
> > want to volunteer to host?  I'd be happy to handle the logistics and
> > host here at Cloudera, but I don't want to take the opportunity away
> > from another company. :)
> >
> > Daniel
> >
> > On 10/28/16 3:27 PM, Vinod Kumar Vavilapalli wrote:
> >> Thanks to everyone who joined this meetup!
> >>
> >> We had quite a blast both in the western hemisphere and from what I
> >> hear in the IST timezone too.
> >>
> >> Overall, stats
> >>
> >>   - PST
> >>  — Started at 269 patch-available tickets and got it down to 170
> >> - a mix of commits, reviews + updates, closing invalid /
> >> not-applicable JIRAs
> >>  — Working notes:
> >>
> https://docs.google.com/spreadsheets/d/1kPKsm3VSnkLU107t-CL05RQ9xQxc6kN-h6o9bDrheaY/edit#gid=2076540402
> >>
> >>   - IST: (Notes from Sunil)
> >>  — 19 patch commits, 11 added / rebased patches, and more commits
> >> on the way waiting for Jenkins
> >>  — Working notes:
> >>
> https://docs.google.com/spreadsheets/d/1EVga79x-sxrfxWoe3o_ZkyLgHbrzHJqAU4hq6qabW_k/edit?ts=5811eb51#gid=0
> >>
> >> Special thanks
> >>   - To Subru for sponsoring the event, logistics and a great lunch!
> >>   - To Sunil for taking the initiative, organizing and running the
> >> contributors’ meetup in India!
> >>
> >> We are thinking of doing this at a regular cadence but for a smaller
> >> duration than a full-day.
> >>
> >> Thanks
> >> +Vinod
> >>
> >>> On Oct 19, 2016, at 11:08 AM, Subru Krishnan  wrote:
> >>>
> >>> Folks,
> >>>
> >>> Hope everyone's is doing great.
> >>>
> >>> We are putting in one full day (5-6 hours) for a YARN review / commit
> >>> marathon on *next Thursday, 27th Oct*.
> >>>
> >>> Expected Audience: *regular contributor / committer in YARN*.
> >>>
> >>> Non-audience: While the meetups are generally open to the
> >>> general
> >>> public, this is not a 'meetup to learn about YARN'.
> >>>
> >>> Specific Agenda: YARN bug bash
> >>>
> >>> Location: Microsoft Moffett Towers, 1020 Enterprise Way,
> >>> Sunnyvale,
> >>> CA.
> >>>
> >>> Webex/Skype details for those who are remote: TBD
> >>>
> >>> Meetup URL:
> >>> http://www.meetup.com/Hadoop-Contributors/events/234971372/
> >>>
> >>> IMPORTANT NOTES:
> >>> - Food will be provided for the reviewers / committers to make sure
> >>> they
> >>> stay up :)
> >>> - We have capacity for only *25 people*, so this isn't a walk-in,
> >>> please *RSVP
> >>> *and reach out to us if you want to join this meetup.
> >>>
> >>> Thanks.
> >>
> >> -
> >> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >>
> >
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (YARN-7488) Make ServiceClient.getAppId method public to return ApplicationId for a service name

2017-11-13 Thread Gour Saha (JIRA) 
Gour Saha created YARN-7488:
---

 Summary: Make ServiceClient.getAppId method public to return 
ApplicationId for a service name
 Key: YARN-7488
 URL: https://issues.apache.org/jira/browse/YARN-7488
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Gour Saha


There are several APIs in YarnClient which can be used when the ApplicationId 
is available. For a YARN service getting the ApplicationId corresponding to a 
service name will open up all these APIs for the service. Hence making the 
below API public will be very helpful.
{code}
  private synchronized ApplicationId getAppId(String serviceName)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop 2.8.3 Release Plan

2017-11-13 Thread Junping Du 
Hi,
We have several important fixes get landed on branch-2.8 and I would like 
to cut off branch-2.8.3 now to start 2.8.3 release work. 
So far, I don't see any pending blockers on 2.8.3, so my current plan is to 
cut off 1st RC of 2.8.3 in next several days: 
 -  For all coming commits to land on branch-2.8, please mark the fix 
version as 2.8.4.
 -  If there is a really important fix for 2.8.3 and getting closed, 
please notify me ahead before landing it on branch-2.8.3.
Please let me know if you have any thoughts or comments on the plan.

Thanks,

Junping

From: dujunp...@gmail.com  on behalf of 俊平堵 

Sent: Friday, October 27, 2017 3:33 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.2 Release.

Hi all,

It gives me great pleasure to announce that the Apache Hadoop community
has
voted to release Apache Hadoop 2.8.2, which is now available for download
from
Apache mirrors[1]. For download instructions please refer to the Apache
Hadoop
Release page [2].

Apache Hadoop 2.8.2 is the first GA release of Apache Hadoop 2.8 line and
our
newest stable release for entire Apache Hadoop project. For major changes
incuded in Hadoop 2.8 line, please refer Hadoop 2.8.2 main page[3].

This release has 315 resolved issues since previous 2.8.1 release with
following
breakdown:
   - 91 in Hadoop Common
   - 99 in HDFS
   - 105 in YARN
   - 20 in MapReduce
Please read the log of CHANGES[4] and RELEASENOTES[5] for more details.

The release news is posted on the Hadoop website too, you can go to the
downloads section directly [6].

Thank you all for contributing to the Apache Hadoop release!


Cheers,

Junping


[1] http://www.apache.org/dyn/closer.cgi/hadoop/common

[2] http://hadoop.apache.org/releases.html

[3] http://hadoop.apache.org/docs/r2.8.2/index.html

[4]
http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/release/2.8.2/CHANGES.2.8.2.html

[5]
http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/release/2.8.2/RELEASENOTES.2.8.2.html

[6] http://hadoop.apache.org/releases.html#Download


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7487) Make sure volume includes GPU base libraries exists after created by plugin

2017-11-13 Thread Wangda Tan (JIRA) 
Wangda Tan created YARN-7487:


 Summary: Make sure volume includes GPU base libraries exists after 
created by plugin
 Key: YARN-7487
 URL: https://issues.apache.org/jira/browse/YARN-7487
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Wangda Tan
Assignee: Wangda Tan


YARN-7224 will create docker volume includes GPU base libraries when launch a 
docker container which needs GPU. 

This JIRA will add necessary checks to make sure docker volume exists before 
launching the container to reduce debug efforts if container fails.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7486) Race condition in service AM that can cause NPE

2017-11-13 Thread Jian He (JIRA) 
Jian He created YARN-7486:
-

 Summary: Race condition in service AM that can cause NPE
 Key: YARN-7486
 URL: https://issues.apache.org/jira/browse/YARN-7486
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] A final minor release off branch-2?

2017-11-13 Thread Wangda Tan 
Thanks Vinod for staring this,

I'm also leaning towards the plan (A):




* (A)-- Make 2.9.x the last minor release off branch-2-- Have a
maintenance release that bridges 2.9 to 3.x-- Continue to make more
maintenance releases on 2.8 and 2.9 as necessary*

The only part I'm not sure is having a separate bridge release other than
3.x.

For the bridge release, Steve's suggestion sounds more doable:

** 3.1+ for new features*
** fixes to 3.0.x &, where appropriate, 2.9, esp feature stabilisation*
** whoever puts their hand up to do 2.x releases deserves support in
testing *
** If someone makes a really strong case to backport a feature from 3.x to
branch-2 and its backwards compatible, I'm not going to stop them. It's
just once 3.0 is out and a 3.1 on the way, it's less compelling*

This makes community can focus on 3.x releases and fill whatever gaps of
migrating from 2.x to 3.x.

Best,
Wangda


On Wed, Nov 8, 2017 at 3:57 AM, Steve Loughran 
wrote:

>
> > On 7 Nov 2017, at 19:08, Vinod Kumar Vavilapalli 
> wrote:
> >
> >
> >
> >
> >> Frankly speaking, working on some bridging release not targeting any
> feature isn't so attractive to me as a contributor. Overall, the final
> minor release off branch-2 is good, we should also give 3.x more time to
> evolve and mature, therefore it looks to me we would have to work on two
> release lines meanwhile for some time. I'd like option C), and suggest we
> focus on the recent releases.
> >
> >
> >
> > Answering this question is also one of the goals of my starting this
> thread. Collectively we need to conclude if we are okay or not okay with no
> longer putting any new feature work in general on the 2.x line after 2.9.0
> release and move over our focus into 3.0.
> >
> >
> > Thanks
> > +Vinod
> >
>
>
> As a developer of new features (e.g the Hadoop S3A committers), I'm mostly
> already committed to targeting 3.1; the code in there to deal with failures
> and retries has unashamedly embraced java 8 lambda-expressions in
> production code: backporting that is going to be traumatic in terms of
> IDE-assisted code changes and the resultant diff in source between branch-2
> & trunk. What's worse, its going to be traumatic to test as all my JVMs
> start with an 8 at the moment, and I'm starting to worry about whether I
> should bump a windows VM up to Java 9 to keep an eye on Akira's work there.
> Currently the only testing I'm really doing on java 7 is yetus branch-2 &
> internal test runs.
>
>
> 3.0 will be out the door, and we can assume that CDH will ship with it
> soon (*)  which will allow for a rapid round trip time on inevitable bugs:
> 3.1 can be the release with compatibility tuned, those reported issues
> addressed. It's certainly where I'd like to focus.
>
>
> At the same time: 2.7.2-2.8.x are the broadly used versions, we can't just
> say "move to 3.0" & expect everyone to do it, not given we have explicitly
> got backwards-incompatible changes in. I don't seen people rushing to do it
> until the layers above are all qualified (HBase, Hive, Spark, ...). Which
> means big users of 2.7/2,8 won't be in a rush to move and we are going to
> have to maintain 2.x for a while, including security patches for old
> versions. One issue there: what if a patch (such as bumping up a JAR
> version) is incompatible?
>
> For me then
>
> * 3.1+ for new features
> * fixes to 3.0.x &, where appropriate, 2.9, esp feature stabilisation
> * whoever puts their hand up to do 2.x releases deserves support in
> testing 
> * If someone makes a really strong case to backport a feature from 3.x to
> branch-2 and its backwards compatible, I'm not going to stop them. It's
> just once 3.0 is out and a 3.1 on the way, it's less compelling
>
> -Steve
>
> Note: I'm implicitly assuming a timely 3.1 out the door with my work
> included, all all issues arriving from 3,0 fixed. We can worry when 3.1
> ships whether there's any benefit in maintaining a 3.0.x, or whether it's
> best to say "move to 3.1"
>
>
>
> (*) just a guess based the effort & test reports of Andrew & others
>
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC2)

2017-11-13 Thread Xiao Chen 
I have no issues now that RC3 is rolling. Thanks for the quick actions.

My point is I'm not sure that's even required. MIT licenses standard format
has that copyright line https://en.wikipedia.org/wiki/MIT_License, and
combined with the licensing-howto I referred earlier, HADOOP-15036 is not
necessary. Though IANAL.

-Xiao

On Mon, Nov 13, 2017 at 3:27 PM, Arun Suresh  wrote:

> Cancelling the RC2 to fix the LICENSE.txt issue.
> Will be rolling out an RC3 shortly. Given that the delta is just the
> LICENSE fix, we will be carrying over the votes from this thread as well.
>
> Cheers
> -Arun/Subru
>
> On Mon, Nov 13, 2017 at 2:59 PM, Arun Suresh  wrote:
>
>> Hi Xiao,
>> What Anu pointed out was that the copyright was missing in the LICENCE
>> not notice. Don't think we need to file LEGAL, since what we are doing is
>> similar to the other MIT dependencies.
>>
>> Cheers
>> -Arun
>>
>>
>>
>> On Mon, Nov 13, 2017 at 2:53 PM, Xiao Chen  wrote:
>>
>>> Thanks guys for pulling the RC and prompt reviews.
>>>
>>> From previous L discussions
>>> >> ommentId=15284739=com.atlassian.jira.plugin.system.issu
>>> etabpanels:comment-tabpanel#comment-15284739>
>>>  and http://www.apache.org/dev/licensing-howto.html#permissive-deps, my
>>> impression was BSD / MIT doesn't require a Notice. (Usually the copyright
>>> parts go to the notice). For this case, the trickiness comes from the
>>> License itself has this added line of copyright...
>>>
>>> Should we file a LEGAL jira to confirm first? There may (or may not) be
>>> other similar instances.
>>>
>>> -Xiao
>>>
>>> On Mon, Nov 13, 2017 at 2:34 PM, Arun Suresh  wrote:
>>>
>>> > Thanks for notifying Anu,
>>> > We've raised https://issues.apache.org/jira/browse/HADOOP-15036 to
>>> address
>>> > this and have included a patch.
>>> > Can you please take a look and possibly +1 if you are fine ?
>>> >
>>> > Cheers
>>> > -Arun
>>> >
>>> > On Mon, Nov 13, 2017 at 2:19 PM, Anu Engineer <
>>> aengin...@hortonworks.com>
>>> > wrote:
>>> >
>>> > > -1 (binding)
>>> > >
>>> > > Thank you for all the hard work on 2.9 series. Unfortunately, this
>>> is one
>>> > > of the times I have to -1 this release.
>>> > >
>>> > > Looks like HADOOP-14840 added a dependency on “oj! Algorithms -
>>> version
>>> > > 43.0”, but we have just added “oj! Algorithms - version 43.0” to the
>>> > > “LICENSE.txt”. The right addition to the LICENESE.txt should contain
>>> the
>>> > > original MIT License, especially “Copyright (c) 2003-2017
>>> Optimatika”.
>>> > >
>>> > > Please take a look at https://github.com/optimatika/
>>> > > ojAlgo/blob/master/LICENSE
>>> > >
>>> > > I am a +1 after this is fixed.
>>> > >
>>> > > Thanks
>>> > > Anu
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On 11/13/17, 9:50 AM, "Sunil G"  wrote:
>>> > >
>>> > > +1 (binding)
>>> > >
>>> > > Deployed cluster built from source.
>>> > >
>>> > >
>>> > >
>>> > >- Tested few cases in an HA cluster and tried to do failover
>>> by
>>> > > using
>>> > >rmadmin commands etc. This seems works fine including
>>> submitting
>>> > > apps.
>>> > >- I also tested many MR apps and all are running fine w/o any
>>> > > issues.
>>> > >- Majorly tested below feature sanity too (works fine)
>>> > >   - Application priority
>>> > >   - Application timeout
>>> > >- Tested basic NodeLabel scenarios.
>>> > >   - Added some labels to couple of nodes
>>> > >   - Verified old UI for labels
>>> > >   - Submitted apps to labelled cluster and it works fine.
>>> > >   - Also performed few cli commands related to nodelabel
>>> > >- Verified new YARN UI and accessed various pages when
>>> cluster was
>>> > > in
>>> > >use. It seems fine to me.
>>> > >
>>> > >
>>> > > Thanks all folks who participated in this release, appreciate the
>>> > same!
>>> > >
>>> > > - Sunil
>>> > >
>>> > >
>>> > > On Mon, Nov 13, 2017 at 3:01 AM Subru Krishnan >> >
>>> > > wrote:
>>> > >
>>> > > > Hi Folks,
>>> > > >
>>> > > > Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and
>>> > will
>>> > > be the
>>> > > > starting release for Apache Hadoop 2.9.x line - it includes 30
>>> New
>>> > > Features
>>> > > > with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed
>>> > issues
>>> > > since
>>> > > > 2.8.2.
>>> > > >
>>> > > > More information about the 2.9.0 release plan can be found
>>> here:
>>> > > > *
>>> > > > https://cwiki.apache.org/confluence/display/HADOOP/
>>> > > Roadmap#Roadmap-Version2.9
>>> > > > <
>>> > > > https://cwiki.apache.org/confluence/display/HADOOP/
>>> > > Roadmap#Roadmap-Version2.9
>>> > > > >*
>>> > > >
>>> > > > New RC is available at: http://home.apache.org/~
>>> > > asuresh/hadoop-2.9.0-RC2/

[VOTE] Release Apache Hadoop 2.9.0 (RC3)

2017-11-13 Thread Arun Suresh 
Hi Folks,

Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
2.8.2.

More information about the 2.9.0 release plan can be found here:
*https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
*

New RC is available at: *https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
*

The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
756ebc8394e473ac25feac05fa493f6d612e6c50.

The maven artifacts are available via repository.apache.org at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1068/
*

We are carrying over the votes from the previous RC given that the delta is
the license fix.

Given the above - we are also going to stick with the original deadline for
the vote : ending on Friday 17th November 2017 2pm PT time.

Thanks,
-Arun/Subru

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC2)

2017-11-13 Thread Arun Suresh 
Cancelling the RC2 to fix the LICENSE.txt issue.
Will be rolling out an RC3 shortly. Given that the delta is just the
LICENSE fix, we will be carrying over the votes from this thread as well.

Cheers
-Arun/Subru

On Mon, Nov 13, 2017 at 2:59 PM, Arun Suresh  wrote:

> Hi Xiao,
> What Anu pointed out was that the copyright was missing in the LICENCE not
> notice. Don't think we need to file LEGAL, since what we are doing is
> similar to the other MIT dependencies.
>
> Cheers
> -Arun
>
>
>
> On Mon, Nov 13, 2017 at 2:53 PM, Xiao Chen  wrote:
>
>> Thanks guys for pulling the RC and prompt reviews.
>>
>> From previous L discussions
>> > ommentId=15284739=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-15284739>
>>  and http://www.apache.org/dev/licensing-howto.html#permissive-deps, my
>> impression was BSD / MIT doesn't require a Notice. (Usually the copyright
>> parts go to the notice). For this case, the trickiness comes from the
>> License itself has this added line of copyright...
>>
>> Should we file a LEGAL jira to confirm first? There may (or may not) be
>> other similar instances.
>>
>> -Xiao
>>
>> On Mon, Nov 13, 2017 at 2:34 PM, Arun Suresh  wrote:
>>
>> > Thanks for notifying Anu,
>> > We've raised https://issues.apache.org/jira/browse/HADOOP-15036 to
>> address
>> > this and have included a patch.
>> > Can you please take a look and possibly +1 if you are fine ?
>> >
>> > Cheers
>> > -Arun
>> >
>> > On Mon, Nov 13, 2017 at 2:19 PM, Anu Engineer <
>> aengin...@hortonworks.com>
>> > wrote:
>> >
>> > > -1 (binding)
>> > >
>> > > Thank you for all the hard work on 2.9 series. Unfortunately, this is
>> one
>> > > of the times I have to -1 this release.
>> > >
>> > > Looks like HADOOP-14840 added a dependency on “oj! Algorithms -
>> version
>> > > 43.0”, but we have just added “oj! Algorithms - version 43.0” to the
>> > > “LICENSE.txt”. The right addition to the LICENESE.txt should contain
>> the
>> > > original MIT License, especially “Copyright (c) 2003-2017 Optimatika”.
>> > >
>> > > Please take a look at https://github.com/optimatika/
>> > > ojAlgo/blob/master/LICENSE
>> > >
>> > > I am a +1 after this is fixed.
>> > >
>> > > Thanks
>> > > Anu
>> > >
>> > >
>> > >
>> > >
>> > > On 11/13/17, 9:50 AM, "Sunil G"  wrote:
>> > >
>> > > +1 (binding)
>> > >
>> > > Deployed cluster built from source.
>> > >
>> > >
>> > >
>> > >- Tested few cases in an HA cluster and tried to do failover by
>> > > using
>> > >rmadmin commands etc. This seems works fine including
>> submitting
>> > > apps.
>> > >- I also tested many MR apps and all are running fine w/o any
>> > > issues.
>> > >- Majorly tested below feature sanity too (works fine)
>> > >   - Application priority
>> > >   - Application timeout
>> > >- Tested basic NodeLabel scenarios.
>> > >   - Added some labels to couple of nodes
>> > >   - Verified old UI for labels
>> > >   - Submitted apps to labelled cluster and it works fine.
>> > >   - Also performed few cli commands related to nodelabel
>> > >- Verified new YARN UI and accessed various pages when cluster
>> was
>> > > in
>> > >use. It seems fine to me.
>> > >
>> > >
>> > > Thanks all folks who participated in this release, appreciate the
>> > same!
>> > >
>> > > - Sunil
>> > >
>> > >
>> > > On Mon, Nov 13, 2017 at 3:01 AM Subru Krishnan 
>> > > wrote:
>> > >
>> > > > Hi Folks,
>> > > >
>> > > > Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and
>> > will
>> > > be the
>> > > > starting release for Apache Hadoop 2.9.x line - it includes 30
>> New
>> > > Features
>> > > > with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed
>> > issues
>> > > since
>> > > > 2.8.2.
>> > > >
>> > > > More information about the 2.9.0 release plan can be found here:
>> > > > *
>> > > > https://cwiki.apache.org/confluence/display/HADOOP/
>> > > Roadmap#Roadmap-Version2.9
>> > > > <
>> > > > https://cwiki.apache.org/confluence/display/HADOOP/
>> > > Roadmap#Roadmap-Version2.9
>> > > > >*
>> > > >
>> > > > New RC is available at: http://home.apache.org/~
>> > > asuresh/hadoop-2.9.0-RC2/
>> > > > <
>> > > > http://www.google.com/url?q=http%3A%2F%2Fhome.apache.org%
>> > > 2F~asuresh%2Fhadoop-2.9.0-RC1%2F=D=1=
>> > > AFQjCNE7BF35IDIMZID3hPqiNglWEVsTpg
>> > > > >
>> > > >
>> > > > The RC tag in git is: release-2.9.0-RC2, and the latest commit
>> id
>> > is:
>> > > > 1eb05c1dd48fbc9e4b375a76f2046a59103bbeb1.
>> > > >
>> > > > The maven artifacts are available via repository.apache.org at:
>> > > > https://repository.apache.org/content/repositories/
>> > > orgapachehadoop-1067/
>> > > > <
>> > > >

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC2)

2017-11-13 Thread Anu Engineer 
-1 (binding)

Thank you for all the hard work on 2.9 series. Unfortunately, this is one of 
the times I have to -1 this release.

Looks like HADOOP-14840 added a dependency on “oj! Algorithms - version 43.0”, 
but we have just added “oj! Algorithms - version 43.0” to the 
“LICENSE.txt”. The right addition to the LICENESE.txt should contain the 
original MIT License, especially “Copyright (c) 2003-2017 Optimatika”.

Please take a look at https://github.com/optimatika/ojAlgo/blob/master/LICENSE

I am a +1 after this is fixed.

Thanks 
Anu




On 11/13/17, 9:50 AM, "Sunil G"  wrote:

+1 (binding)

Deployed cluster built from source.



   - Tested few cases in an HA cluster and tried to do failover by using
   rmadmin commands etc. This seems works fine including submitting apps.
   - I also tested many MR apps and all are running fine w/o any issues.
   - Majorly tested below feature sanity too (works fine)
  - Application priority
  - Application timeout
   - Tested basic NodeLabel scenarios.
  - Added some labels to couple of nodes
  - Verified old UI for labels
  - Submitted apps to labelled cluster and it works fine.
  - Also performed few cli commands related to nodelabel
   - Verified new YARN UI and accessed various pages when cluster was in
   use. It seems fine to me.


Thanks all folks who participated in this release, appreciate the same!

- Sunil


On Mon, Nov 13, 2017 at 3:01 AM Subru Krishnan  wrote:

> Hi Folks,
>
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be 
the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New 
Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
>
> More information about the 2.9.0 release plan can be found here:
> *
> 
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> <
> 
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> >*
>
> New RC is available at: http://home.apache.org/~asuresh/hadoop-2.9.0-RC2/
> <
> 
http://www.google.com/url?q=http%3A%2F%2Fhome.apache.org%2F~asuresh%2Fhadoop-2.9.0-RC1%2F=D=1=AFQjCNE7BF35IDIMZID3hPqiNglWEVsTpg
> >
>
> The RC tag in git is: release-2.9.0-RC2, and the latest commit id is:
> 1eb05c1dd48fbc9e4b375a76f2046a59103bbeb1.
>
> The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1067/
> <
> 
https://www.google.com/url?q=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg
> >
>
> Please try the release and vote; the vote will run for the usual 5 days,
> ending on Friday 17th November 2017 2pm PT time.
>
> We want to give a big shout out to Sunil, Varun, Rohith, Wangda, Vrushali
> and Inigo for the extensive testing/validation which helped prepare for
> RC2. Do report your results in this vote as it'll be very useful to the
> entire community.
>
> Thanks,
> -Subru/Arun
>

[jira] [Created] (YARN-7485) Add LOG.isDebugEnabled() guard for LOG.debug("")

2017-11-13 Thread Mehran Hassani (JIRA) 
Mehran Hassani created YARN-7485:


 Summary: Add LOG.isDebugEnabled() guard for LOG.debug("")
 Key: YARN-7485
 URL: https://issues.apache.org/jira/browse/YARN-7485
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Mehran Hassani
Priority: Minor


I am conducting research on log related bugs. I tried to make a tool to fix 
repetitive yet simple patterns of bugs that are related to logs. In these 
files, there are debug level logging statements containing multiple string 
concatenation without the if statement before them: 

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/HadoopYarnProtoRPC.java,
 LOG.debug("Creating a HadoopYarnProtoRpc server for protocol " + protocol +" 
with " + numHandlers + " handlers");, 61
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java,
 LOG.debug("Received node update event:" + type + " for node:" + node+ " with 
state:" + nodeState);, 985

Would you be interested in adding the if before these logging statements?

There are 14 more cases that I can mention if needed. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC2)

2017-11-13 Thread Sunil G 
+1 (binding)

Deployed cluster built from source.



   - Tested few cases in an HA cluster and tried to do failover by using
   rmadmin commands etc. This seems works fine including submitting apps.
   - I also tested many MR apps and all are running fine w/o any issues.
   - Majorly tested below feature sanity too (works fine)
  - Application priority
  - Application timeout
   - Tested basic NodeLabel scenarios.
  - Added some labels to couple of nodes
  - Verified old UI for labels
  - Submitted apps to labelled cluster and it works fine.
  - Also performed few cli commands related to nodelabel
   - Verified new YARN UI and accessed various pages when cluster was in
   use. It seems fine to me.


Thanks all folks who participated in this release, appreciate the same!

- Sunil


On Mon, Nov 13, 2017 at 3:01 AM Subru Krishnan  wrote:

> Hi Folks,
>
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
>
> More information about the 2.9.0 release plan can be found here:
> *
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> <
> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> >*
>
> New RC is available at: http://home.apache.org/~asuresh/hadoop-2.9.0-RC2/
> <
> http://www.google.com/url?q=http%3A%2F%2Fhome.apache.org%2F~asuresh%2Fhadoop-2.9.0-RC1%2F=D=1=AFQjCNE7BF35IDIMZID3hPqiNglWEVsTpg
> >
>
> The RC tag in git is: release-2.9.0-RC2, and the latest commit id is:
> 1eb05c1dd48fbc9e4b375a76f2046a59103bbeb1.
>
> The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1067/
> <
> https://www.google.com/url?q=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066=D=1=AFQjCNFcern4uingMV_sEreko_zeLlgdlg
> >
>
> Please try the release and vote; the vote will run for the usual 5 days,
> ending on Friday 17th November 2017 2pm PT time.
>
> We want to give a big shout out to Sunil, Varun, Rohith, Wangda, Vrushali
> and Inigo for the extensive testing/validation which helped prepare for
> RC2. Do report your results in this vote as it'll be very useful to the
> entire community.
>
> Thanks,
> -Subru/Arun
>

[jira] [Created] (YARN-7483) Test cases cleanup post YARN-5881

2017-11-13 Thread Sunil G (JIRA) 
Sunil G created YARN-7483:
-

 Summary: Test cases cleanup post YARN-5881
 Key: YARN-7483
 URL: https://issues.apache.org/jira/browse/YARN-7483
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: test
Affects Versions: YARN-5881
Reporter: Sunil G
Assignee: Sunil G


Few UT cases has to be cleaned or rewritten cleanly to consider both 
absolute/percentage based configuration.

Related test classes
# TestLeafQueue
# TestCapacityScheduler



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7481) Gpu locality support for Better AI scheduling

2017-11-13 Thread Chen Qingcha (JIRA) 
Chen Qingcha created YARN-7481:
--

 Summary: Gpu locality support for Better AI scheduling
 Key: YARN-7481
 URL: https://issues.apache.org/jira/browse/YARN-7481
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: api, RM, yarn
Affects Versions: 2.7.2
Reporter: Chen Qingcha
 Fix For: 2.7.2


We enhance Hadoop with GPU support for better AI job scheduling. 
Currently, YARN-3926 also supports GPU scheduling, which treats GPU as 
countable resource. 
However, GPU placement is very important to deep learning job for better 
efficiency.
 For example, a 2-GPU job runs on gpu {0,1} could be faster than run on gpu {0, 
7}, if GPU 0 and 1 are under the same PCI-E switch while 0 and 7 are not.
 We add the GPU support to Hadoop 2.7.2 to enable GPU locality scheduling, 
which support fine-grained GPU placement. A 64-bits bitmap is added to yarn 
Resource, which indicates both GPU usage and locality information in a node (up 
to 64 GPUs per node). '1' means available and '0' otherwise in the 
corresponding position of the bit.   



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7480) Render tooltips on columns where text is clipped

2017-11-13 Thread Vasudevan Skm (JIRA) 
Vasudevan Skm created YARN-7480:
---

 Summary: Render tooltips on columns where text is clipped
 Key: YARN-7480
 URL: https://issues.apache.org/jira/browse/YARN-7480
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn-ui-v2
Reporter: Vasudevan Skm
Assignee: Vasudevan Skm


In em-table, when text gets clipped the information is lost. Need to render a 
tooltip to show the full text in these cases



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org