date:20170830

[jira] [Created] (YARN-7139) FairScheduler: finished applications are always restored to default queue

2017-08-30 Thread Wilfred Spiegelenburg (JIRA)

Wilfred Spiegelenburg created YARN-7139:
---

 Summary: FairScheduler: finished applications are always restored 
to default queue
 Key: YARN-7139
 URL: https://issues.apache.org/jira/browse/YARN-7139
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler
Affects Versions: 2.8.1
Reporter: Wilfred Spiegelenburg
Assignee: Wilfred Spiegelenburg


The queue an application gets submitted to is defined by the placement policy 
in the FS. The placement policy returns the queue and the application object is 
updated. When an application is stored in the state store the application 
submission context is used which has not been updated after the placement rules 
have run. 

This means that the original queue from the submission is still stored which is 
the incorrect queue. On restore we then read back the wrong queue and display 
the wrong queue in the RM web UI.

We should update the submission context after we have run the placement 
policies to make sure that we store the correct queue for the application.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7138) Fix incompatible API change for YarnScheduler involved by YARN-5521

2017-08-30 Thread Junping Du (JIRA)

Junping Du created YARN-7138:


 Summary: Fix incompatible API change for YarnScheduler involved by 
YARN-5521
 Key: YARN-7138
 URL: https://issues.apache.org/jira/browse/YARN-7138
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler
Reporter: Junping Du
Priority: Blocker


>From JACC report for 2.8.2 against 2.7.4, it indicates that we have 
>incompatible changes happen in YarnScheduler:
{noformat}
hadoop-yarn-server-resourcemanager-2.7.4.jar, YarnScheduler.class
package org.apache.hadoop.yarn.server.resourcemanager.scheduler
YarnScheduler.allocate ( ApplicationAttemptId p1, List p2, 
List p3, List p4, List p5 ) [abstract]  :  
Allocation 
{noformat}
The root cause is YARN-5221. We should change it back or workaround this by 
adding back original API (mark as deprecated if not used any more).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7137) Resource profile: Move new added APIs to unstable

2017-08-30 Thread Wangda Tan (JIRA)

Wangda Tan created YARN-7137:


 Summary: Resource profile: Move new added APIs to unstable
 Key: YARN-7137
 URL: https://issues.apache.org/jira/browse/YARN-7137
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Wangda Tan
Assignee: Wangda Tan
Priority: Blocker






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-30 Thread Vrushali C

Thanks Sangjin for the link to the previous discussions on this! I think
that helps answer Steve's questions.

As decided on that thread [1], YARN-5355 as a feature branch was merged to
trunk via "git merge --no-ff" .

Although trunk already had TSv2 code (alpha1) prior to this merge, we chose
to develop on a feature branch YARN-5355 so that we could control when
changes went into trunk and didn't inadvertently disrupt trunk.

Is the latest merge causing any conflicts or issues for s3guard, Steve?

thanks
Vrushali
[1]
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E


On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee  wrote:

> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran 
> wrote:
>
>> I'd have assumed it would have gone in as one single patch, rather than a
>> full history. I don't see why the trunk needs all the evolutionary history
>> of a build.
>>
>> What should our policy/process be here?
>>
>> I do currently plan to merge the s3guard in as one single squashed patch;
>> just getting HADOOP-14809 sorted first.
>>
>>
>> > On 30 Aug 2017, at 07:09, Vrushali C  wrote:
>> >
>> > I'm adding my +1 (binding) to conclude the vote.
>> >
>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>> with
>> > the merge to trunk shortly. Thanks everyone!
>> >
>> > Regards
>> > Vrushali
>> >
>> >
>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
>> > varun.saxena.apa...@gmail.com> wrote:
>> >
>> >> +1 (binding).
>> >>
>> >> Kudos to all the team members for their great work!
>> >>
>> >> Being part of the ATSv2 team, I have been involved with either
>> development
>> >> or review of most of the JIRAs'.
>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>> there
>> >> is no impact when ATSv2 is turned off.
>> >>
>> >> Regards,
>> >> Varun Saxena.
>> >>
>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> >> vrushalic2...@gmail.com> wrote:
>> >>
>> >>> Hi folks,
>> >>>
>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> >>> will
>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> >>>
>> >>> We have previously completed one merge onto trunk [3] and Timeline
>> Service
>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> >>>
>> >>> Since then, we have been working on extending the capabilities of
>> Timeline
>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> >>> confident that the state of the feature meets the criteria to be
>> merged
>> >>> onto trunk and we'd love folks to get their hands on it in a test
>> capacity
>> >>> and provide valuable feedback so that we can make it production-ready.
>> >>>
>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>> and
>> >>> usability improvements based on a new architecture. What we would
>> like to
>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> >>> complete end-to-end read/write flow with security and read level
>> >>> authorization via whitelists. You should be able to start setting it
>> up
>> >>> and
>> >>> testing it.
>> >>>
>> >>> At a high level, the following are the key features that have been
>> >>> implemented since alpha1:
>> >>> - Security via Kerberos Authentication and delegation tokens
>> >>> - Read side simple authorization via whitelist
>> >>> - Client configurable entity sort ordering
>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> metrics by
>> >>> timerange, pagination, sub-app entities
>> >>> - Support for storing sub-application entities (entities that exist
>> >>> outside
>> >>> the scope of an application)
>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
>> >>> prefixes,
>> >>> configurable hbase cluster
>> >>> - Flow level aggregations done as dynamic (table level) coprocessors
>> >>> - Uses latest stable HBase release 1.2.6
>> >>>
>> >>> There are a total of 82 subtasks that were completed as part of this
>> >>> effort.
>> >>>
>> >>> We paid close attention to ensure that once disabled Timeline Service
>> v.2
>> >>> does not impact existing functionality when disabled (by default).
>> >>>
>> >>> Special thanks to a team of folks who worked hard and contributed
>> towards
>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
>> Varun
>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> >>>
>> >>> Regards,
>> >>> Vrushali
>> >>>
>> >>> [1]

[jira] [Created] (YARN-7136) Additional Performance Improvement for Resource Profile Feature

2017-08-30 Thread Wangda Tan (JIRA)

Wangda Tan created YARN-7136:


 Summary: Additional Performance Improvement for Resource Profile 
Feature
 Key: YARN-7136
 URL: https://issues.apache.org/jira/browse/YARN-7136
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Wangda Tan
Assignee: Wangda Tan
Priority: Critical


This JIRA is plan to add following misc perf improvements:

1) Use final int in Resources/ResourceCalculator to cache 
#known-resource-types. (Significant improvement).
2) Catch Java's ArrayOutOfBound Exception instead of checking array.length 
every time. (Significant improvement).
3) Avoid setUnit validation (which is a HashSet lookup) when initialize default 
Memory/VCores ResourceInformation (Significant improvement).
4) Avoid unnecessary loop array in Resource#toString/hashCode. (Some 
improvement).
5) Removed readOnlyResources in BaseResource. (Minor improvement).
6) Removed enum: MandatoryResources, use final integer instead. (Minor 
improvement).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7135) Clean up lock-try order in common scheduler code

2017-08-30 Thread Daniel Templeton (JIRA)

Daniel Templeton created YARN-7135:
--

 Summary: Clean up lock-try order in common scheduler code
 Key: YARN-7135
 URL: https://issues.apache.org/jira/browse/YARN-7135
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: scheduler
Affects Versions: 3.0.0-alpha4
Reporter: Daniel Templeton


There are many places that follow the pattern:{code}try {
  lock.lock();
  ...
} finally {
  lock.unlock();
}{code}

There are a couple of reasons that's a bad idea.  The correct pattern 
is:{code}lock.lock();
try {
  ...
} finally {
  lock.unlock();
}{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7134) AppSchedulingInfo has a dependency on capacity scheduler

2017-08-30 Thread Daniel Templeton (JIRA)

Daniel Templeton created YARN-7134:
--

 Summary: AppSchedulingInfo has a dependency on capacity scheduler
 Key: YARN-7134
 URL: https://issues.apache.org/jira/browse/YARN-7134
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler
Affects Versions: 3.0.0-alpha4
Reporter: Daniel Templeton
Priority: Blocker


The common scheduling code should be independent of all scheduler 
implementations.  YARN-6040 introduced capacity scheduler's {{SchedulingMode}} 
into {{AppSchedulingInfo}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-30 Thread Colin McCabe

The "git" way of doing things would be to rebase the feature branch on
master (trunk) and then commit the patch stack.

Squashing the entire feature into a 10 MB megapatch is the "svn" way of
doing things.

The svn workflow evolved because merging feature branches back to trunk
was really painful in svn.  So people preferred just to basically do an
rsync from a checkout of the feature branch, edit that up a bit to make
sure they weren't overwriting something that happened in trunk, and then
do an "svn commit" that did not tie back to the feature branch.

P.S. Merges were so painful in SVN that some of the organizations I
worked for maintained spreadsheets of patches which had been merged to
each branch, since svn was of so little help with merging

P.P.S. svn eventually got slightly smarter about merges -- I think in
svn 1.6 or something?

Colin


On Wed, Aug 30, 2017, at 14:37, Sangjin Lee wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
> 
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran 
> wrote:
> 
> > I'd have assumed it would have gone in as one single patch, rather than a
> > full history. I don't see why the trunk needs all the evolutionary history
> > of a build.
> >
> > What should our policy/process be here?
> >
> > I do currently plan to merge the s3guard in as one single squashed patch;
> > just getting HADOOP-14809 sorted first.
> >
> >
> > > On 30 Aug 2017, at 07:09, Vrushali C  wrote:
> > >
> > > I'm adding my +1 (binding) to conclude the vote.
> > >
> > > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > > the merge to trunk shortly. Thanks everyone!
> > >
> > > Regards
> > > Vrushali
> > >
> > >
> > > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> > > varun.saxena.apa...@gmail.com> wrote:
> > >
> > >> +1 (binding).
> > >>
> > >> Kudos to all the team members for their great work!
> > >>
> > >> Being part of the ATSv2 team, I have been involved with either
> > development
> > >> or review of most of the JIRAs'.
> > >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> > there
> > >> is no impact when ATSv2 is turned off.
> > >>
> > >> Regards,
> > >> Varun Saxena.
> > >>
> > >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >> vrushalic2...@gmail.com> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> > >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > >>> will
> > >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>>
> > >>> We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > Timeline
> > >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> > >>> confident that the state of the feature meets the criteria to be merged
> > >>> onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > >>> and provide valuable feedback so that we can make it production-ready.
> > >>>
> > >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > >>> usability improvements based on a new architecture. What we would like
> > to
> > >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > >>> complete end-to-end read/write flow with security and read level
> > >>> authorization via whitelists. You should be able to start setting it up
> > >>> and
> > >>> testing it.
> > >>>
> > >>> At a high level, the following are the key features that have been
> > >>> implemented since alpha1:
> > >>> - Security via Kerberos Authentication and delegation tokens
> > >>> - Read side simple authorization via whitelist
> > >>> - Client configurable entity sort ordering
> > >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics by
> > >>> timerange, pagination, sub-app entities
> > >>> - Support for storing sub-application entities (entities that exist
> > >>> outside
> > >>> the scope of an application)
> > >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> prefixes,
> > >>> configurable hbase cluster
> > >>> - Flow level aggregations done as dynamic (table level) coprocessors
> > >>> - Uses latest stable HBase release 1.2.6
> > >>>
> > >>> There are a total of 82 subtasks that were completed as part of this
> > >>> effort.
> > >>>
> > >>> We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > >>> does not impact existing functionality when disabled (by default).
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort with patches, reviews and guidance:

[jira] [Created] (YARN-7133) Clean up lock-try order fair scheduler

2017-08-30 Thread Daniel Templeton (JIRA)

Daniel Templeton created YARN-7133:
--

 Summary: Clean up lock-try order fair scheduler
 Key: YARN-7133
 URL: https://issues.apache.org/jira/browse/YARN-7133
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: fairscheduler
Affects Versions: 3.0.0-alpha4
Reporter: Daniel Templeton


There are many places that follow the pattern:{code}try {
  lock.lock();
  ...
} finally {
  lock.unlock();
}{code}

There are a couple of reasons that's a bad idea.  The correct pattern 
is:{code}lock.lock();
try {
  ...
} finally {
  lock.unlock();
}{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-30 Thread Sangjin Lee

I recall this discussion about a couple of years ago:
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E

On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran 
wrote:

> I'd have assumed it would have gone in as one single patch, rather than a
> full history. I don't see why the trunk needs all the evolutionary history
> of a build.
>
> What should our policy/process be here?
>
> I do currently plan to merge the s3guard in as one single squashed patch;
> just getting HADOOP-14809 sorted first.
>
>
> > On 30 Aug 2017, at 07:09, Vrushali C  wrote:
> >
> > I'm adding my +1 (binding) to conclude the vote.
> >
> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > the merge to trunk shortly. Thanks everyone!
> >
> > Regards
> > Vrushali
> >
> >
> > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> > varun.saxena.apa...@gmail.com> wrote:
> >
> >> +1 (binding).
> >>
> >> Kudos to all the team members for their great work!
> >>
> >> Being part of the ATSv2 team, I have been involved with either
> development
> >> or review of most of the JIRAs'.
> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> there
> >> is no impact when ATSv2 is turned off.
> >>
> >> Regards,
> >> Varun Saxena.
> >>
> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >> vrushalic2...@gmail.com> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> >>> will
> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>>
> >>> We have previously completed one merge onto trunk [3] and Timeline
> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> Timeline
> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> >>> confident that the state of the feature meets the criteria to be merged
> >>> onto trunk and we'd love folks to get their hands on it in a test
> capacity
> >>> and provide valuable feedback so that we can make it production-ready.
> >>>
> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> >>> usability improvements based on a new architecture. What we would like
> to
> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> complete end-to-end read/write flow with security and read level
> >>> authorization via whitelists. You should be able to start setting it up
> >>> and
> >>> testing it.
> >>>
> >>> At a high level, the following are the key features that have been
> >>> implemented since alpha1:
> >>> - Security via Kerberos Authentication and delegation tokens
> >>> - Read side simple authorization via whitelist
> >>> - Client configurable entity sort ordering
> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> metrics by
> >>> timerange, pagination, sub-app entities
> >>> - Support for storing sub-application entities (entities that exist
> >>> outside
> >>> the scope of an application)
> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> prefixes,
> >>> configurable hbase cluster
> >>> - Flow level aggregations done as dynamic (table level) coprocessors
> >>> - Uses latest stable HBase release 1.2.6
> >>>
> >>> There are a total of 82 subtasks that were completed as part of this
> >>> effort.
> >>>
> >>> We paid close attention to ensure that once disabled Timeline Service
> v.2
> >>> does not impact existing functionality when disabled (by default).
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Regards,
> >>> Vrushali
> >>>
> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27383.html
> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> >>>
> >>
> >>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (YARN-7132) FairScheduler.initScheduler() contains a surprising unary plus

2017-08-30 Thread Daniel Templeton (JIRA)

Daniel Templeton created YARN-7132:
--

 Summary: FairScheduler.initScheduler() contains a surprising unary 
plus
 Key: YARN-7132
 URL: https://issues.apache.org/jira/browse/YARN-7132
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: fairscheduler
Affects Versions: 3.0.0-alpha4
Reporter: Daniel Templeton
Priority: Minor


The method contains the following code:{code}
LOG.warn(FairSchedulerConfiguration.UPDATE_INTERVAL_MS
+ " is invalid, so using default value "
+ +FairSchedulerConfiguration.DEFAULT_UPDATE_INTERVAL_MS
+ " ms instead");{code}

Note the beginning of the third line.  One of those plusses should be deleted 
so that no one else spends cycles trying to understand why it even compiles.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-30 Thread Steve Loughran

I'd have assumed it would have gone in as one single patch, rather than a full 
history. I don't see why the trunk needs all the evolutionary history of a 
build.

What should our policy/process be here?

I do currently plan to merge the s3guard in as one single squashed patch; just 
getting HADOOP-14809 sorted first.


> On 30 Aug 2017, at 07:09, Vrushali C  wrote:
> 
> I'm adding my +1 (binding) to conclude the vote.
> 
> With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> the merge to trunk shortly. Thanks everyone!
> 
> Regards
> Vrushali
> 
> 
> On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> varun.saxena.apa...@gmail.com> wrote:
> 
>> +1 (binding).
>> 
>> Kudos to all the team members for their great work!
>> 
>> Being part of the ATSv2 team, I have been involved with either development
>> or review of most of the JIRAs'.
>> Tested ATSv2 in both secure and non-secure mode. Also verified that there
>> is no impact when ATSv2 is turned off.
>> 
>> Regards,
>> Varun Saxena.
>> 
>> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> vrushalic2...@gmail.com> wrote:
>> 
>>> Hi folks,
>>> 
>>> Per earlier discussion [1], I'd like to start a formal vote to merge
>>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>>> will
>>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> 
>>> We have previously completed one merge onto trunk [3] and Timeline Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> 
>>> Since then, we have been working on extending the capabilities of Timeline
>>> Service v2 in a feature branch [2] for a while, and we are reasonably
>>> confident that the state of the feature meets the criteria to be merged
>>> onto trunk and we'd love folks to get their hands on it in a test capacity
>>> and provide valuable feedback so that we can make it production-ready.
>>> 
>>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>>> usability improvements based on a new architecture. What we would like to
>>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>>> complete end-to-end read/write flow with security and read level
>>> authorization via whitelists. You should be able to start setting it up
>>> and
>>> testing it.
>>> 
>>> At a high level, the following are the key features that have been
>>> implemented since alpha1:
>>> - Security via Kerberos Authentication and delegation tokens
>>> - Read side simple authorization via whitelist
>>> - Client configurable entity sort ordering
>>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>>> timerange, pagination, sub-app entities
>>> - Support for storing sub-application entities (entities that exist
>>> outside
>>> the scope of an application)
>>> - Configurable TTLs (time-to-live) for tables, configurable table
>>> prefixes,
>>> configurable hbase cluster
>>> - Flow level aggregations done as dynamic (table level) coprocessors
>>> - Uses latest stable HBase release 1.2.6
>>> 
>>> There are a total of 82 subtasks that were completed as part of this
>>> effort.
>>> 
>>> We paid close attention to ensure that once disabled Timeline Service v.2
>>> does not impact existing functionality when disabled (by default).
>>> 
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> 
>>> Regards,
>>> Vrushali
>>> 
>>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> 
>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7131) FSDownload.unpack should read determine the type of resource by reading the header bytes

2017-08-30 Thread Brook Zhou (JIRA)

Brook Zhou created YARN-7131:


 Summary: FSDownload.unpack should read determine the type of 
resource by reading the header bytes
 Key: YARN-7131
 URL: https://issues.apache.org/jira/browse/YARN-7131
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Brook Zhou
Assignee: Brook Zhou


Currently, there are naive string checks to determine if a resource of a 
particular type (jar, zip, tar.gz) 

There can be cases where this does not work - e.g., the user decides to split 
up a large zip resource as {file1}.zip.001, {file1}.zip.002.

Instead, FSDownload.unpack should read the file header bytes to determine the 
file type.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7130) ATSv2 documentation changes post merge

2017-08-30 Thread Varun Saxena (JIRA)

Varun Saxena created YARN-7130:
--

 Summary: ATSv2 documentation changes post merge
 Key: YARN-7130
 URL: https://issues.apache.org/jira/browse/YARN-7130
 Project: Hadoop YARN
  Issue Type: Bug
  Components: timelineserver
Reporter: Varun Saxena






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7129) Application Catalog for YARN applications

2017-08-30 Thread Eric Yang (JIRA)

Eric Yang created YARN-7129:
---

 Summary: Application Catalog for YARN applications
 Key: YARN-7129
 URL: https://issues.apache.org/jira/browse/YARN-7129
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: applications
Affects Versions: 3.0.0-beta1
Reporter: Eric Yang
Assignee: Eric Yang


YARN native services provides web services API to improve usability of 
application deployment on Hadoop using collection of docker images.  It would 
be nice to have an application catalog system which provides an editorial and 
search interface for YARN applications.  This improves usability of YARN for 
manage the life cycle of applications.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-08-30 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/508/

[Aug 29, 2017 11:21:14 AM] (sunilg) YARN-6386. Show decommissioning nodes in 
new YARN UI. Contributed by
[Aug 29, 2017 11:36:22 AM] (aajisaka) HADOOP-14671. Upgrade to Apache Yetus 
0.5.0.
[Aug 29, 2017 2:52:24 PM] (jlowe) YARN-5816.
[Aug 29, 2017 6:02:43 PM] (stevel) HADOOP-14802. Add support for using 
container saskeys for all accesses.
[Aug 29, 2017 6:03:44 PM] (stevel) HADOOP-14583. wasb throws an exception if 
you try to create a file and
[Aug 29, 2017 6:27:47 PM] (manojpec) HDFS-11912. Snapshot functionality test 
along with randomized file IO
[Aug 29, 2017 9:53:09 PM] (Carlo Curino) YARN-7010. Federation: routing REST 
invocations transparently to
[Aug 29, 2017 10:16:23 PM] (junping_du) YARN-7037. Optimize data transfer with 
zero-copy approach for
[Aug 29, 2017 10:29:22 PM] (wangda) YARN-5219. When an export var command fails 
in launch_container.sh, the
[Aug 29, 2017 11:33:51 PM] (yufei) YARN-7077. TestAMSimulator and 
TestNMSimulator fail. (Contributed by
[Aug 29, 2017 11:57:03 PM] (yzhang) HDFS-12191. Provide option to not capture 
the accessTime change of a
[Aug 30, 2017 3:35:53 AM] (jianhe) YARN-6756. 
ContainerRequest#executionTypeRequest causes NPE. Contributed
[Aug 30, 2017 4:30:02 AM] (xiao) HDFS-12336. Listing encryption zones still 
fails when deleted EZ is not
[Aug 30, 2017 5:15:11 AM] (rohithsharmaks) YARN-6982. Potential issue on 
setting AMContainerSpec#tokenConf to null
[Aug 30, 2017 5:59:50 AM] (varunsaxena) YARN-5156. YARN_CONTAINER_FINISHED of 
YARN_CONTAINERs will always have
[Aug 30, 2017 5:59:50 AM] (varunsaxena) YARN-5229. Refactor 
#isApplicationEntity and #getApplicationEvent from
[Aug 30, 2017 5:59:50 AM] (varunsaxena) YARN-5638. Introduce a collector 
timestamp to uniquely identify
[Aug 30, 2017 5:59:50 AM] (varunsaxena) YARN-3649. Allow configurable prefix 
for hbase table names like prod,
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5715. Introduce entity prefix for 
return and sort order.
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5265. Make HBase configuration for 
the timeline service
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-3359. Recover collector list when 
RM fails over (Li Lu via Varun
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-3359. Addendum for Recover 
collector list when RM fails over (Li Lu
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5792. Adopt the id prefix for 
YARN, MR, and DS entities.
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5739. Provide timeline reader API 
to list available timeline entity
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5922. Remove direct references of 
HBaseTimelineWriter/Reader in
[Aug 30, 2017 5:59:51 AM] (varunsaxena) MAPREDUCE-6818. Remove direct reference 
to TimelineClientImpl.
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5925. Extract 
hbase-backend-exclusive utility methods from
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5585. [Atsv2] Reader side changes 
for entity prefix and support for
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5980. Update documentation for 
single node hbase deploy.
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-5378. Accommodate app-id->cluster 
mapping (Sangjin Lee via Varun
[Aug 30, 2017 5:59:51 AM] (varunsaxena) YARN-6064. Support fromId for flowRuns 
and flow/flowRun apps REST API's
[Aug 30, 2017 5:59:52 AM] (varunsaxena) Addendum for YARN-6064. Support fromId 
for flowRuns and flow/flowRun
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6094. Update the coprocessor to be 
a dynamically loaded one.
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-5928. Move ATSv2 HBase backend 
code into a new module that is only
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-4675. Reorganize TimelineClient 
and TimelineClientImpl into
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6069. CORS support in timeline v2 
(Rohith Sharma K S via Varun
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6253. 
FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6027. Support fromid(offset) 
filter for /flows API (Rohith Sharma K
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6256. Add FROM_ID info key for 
timeline entities in reader response
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6237. Move UID constant to 
TimelineReaderUtils (Rohith Sharma K S
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6318. timeline service schema 
creator fails if executed from a
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6146. Add Builder methods for 
TimelineEntityFilters (Haibo Chen via
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6555. Store application flow 
context in NM state store for
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6658. Remove columnFor() methods 
of Columns in HBaseTimeline
[Aug 30, 2017 5:59:52 AM] (varunsaxena) YARN-6316 Provide help information and 
documentation for
[Aug 30, 2017 5:59:53 AM] (varunsaxena) YARN-5647. [ATSv2 Security] Collector 
side changes for loading auth

[jira] [Resolved] (YARN-7111) ApplicationHistoryServer webpage startTime and state are not readable

2017-08-30 Thread Prabhu Joseph (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prabhu Joseph resolved YARN-7111.
-
Resolution: Not A Problem

> ApplicationHistoryServer webpage startTime and state are not readable
> -
>
> Key: YARN-7111
> URL: https://issues.apache.org/jira/browse/YARN-7111
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: timelineserver
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
> Attachments: Screen Shot 2017-08-28 at 5.24.01 PM.png, working.png
>
>
> ApplicationHistoryServer webpage FINISHED applications displays startTime and 
> state in not readable format. (attached image)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7128) The error message in TimelineSchemaCreator is not enough to find out the error.

2017-08-30 Thread Jinjiang Ling (JIRA)

Jinjiang Ling created YARN-7128:
---

 Summary: The error message in TimelineSchemaCreator is not enough 
to find out the error.
 Key: YARN-7128
 URL: https://issues.apache.org/jira/browse/YARN-7128
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Jinjiang Ling
Assignee: Jinjiang Ling


When I  run the schema creator tool of timeline service v2, then an error 
happened with the error message below:
{quote}
17/08/30 16:04:18 INFO zookeeper.ClientCnxn: Session establishment complete on 
server zdh175/10.43.183.175:2181, sessionid = 0x15e0d065a330019, negotiated 
timeout = 18
17/08/30 16:04:18 WARN util.HeapMemorySizeUtil: 
hbase.regionserver.global.memstore.upperLimit is deprecated by 
hbase.regionserver.global.memstore.size
17/08/30 16:04:18 ERROR storage.TimelineSchemaCreator: Error in creating hbase 
tables: java.lang.reflect.InvocationTargetException
17/08/30 16:04:18 WARN storage.TimelineSchemaCreator: Schema creation finished 
with the following exceptions
17/08/30 16:04:18 WARN storage.TimelineSchemaCreator: 
java.lang.reflect.InvocationTargetException
{quote}
But I can't tell which error caused this exception with message.

Then I find it only print the exception message in TimelineSchemaCreator
{code}
  try {
..
  } catch (IOException e) {
  LOG.error("Error in creating hbase tables: " + e.getMessage());
  exceptions.add(e);
  }
{code}
I change it like this
{code}
  try {
..
  } catch (IOException e) {
  LOG.error("Error in creating hbase tables: ", e);
  exceptions.add(e);
  }
{code}

finally, I get the detail of this error, and it's caused by a directory of 
hbase not created.
{quote}
at 
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:648)
... 10 more
Caused by: java.lang.RuntimeException: Failed to create local dir 
/data1/zdh/hbase/tmp/RegionServer/local/jars, DynamicClassLoader failed to init
at 
org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
at 
org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
at 
org.apache.hadoop.hbase.protobuf.ProtobufUtil.(ProtobufUtil.java:244)
... 15 more
17/08/30 16:03:42 WARN storage.TimelineSchemaCreator: Schema creation finished 
with the following exceptions
17/08/30 16:03:42 WARN storage.TimelineSchemaCreator: 
java.lang.reflect.InvocationTargetException
{quote}

As the exception message is not enough to find out the error, so I think print 
all the information of the exception in TimelineSchemaCreator is better.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

2017-08-30 Thread Junping Du

Thanks Brahma for comment on this thread. To be clear, I always update branch 
version just before RC kicking off.

For 2.8.2 release, I don't have plan to involve big top or other third-party 
test tools. As always, we will rely on test/verify efforts from community 
especially from large deployed production cluster - as far as I know,  there 
are already several companies. like: Yahoo!, Alibaba, etc. already deploy 2.8 
release in large production clusters for months which give me more confidence 
on 2.8.2.


Here is more update on 2.8.2 release:

Blocker issues:

-  A new blocker YARN-7076 get reported and fixed by Jian He through last 
weekend.

-  Another new blocker - HADOOP-14814 get identified from my latest jdiff 
run against 2.7.4. The simple fix on an incompatible API change should get 
commit soon.


Critical issues:

-  YARN-7083 already get committed. Thanks Jason for reporting the issue 
and delivering the fix.

-  YARN-6091 get push out from 2.8.2 as issue is not a regression and 
pending for a while.

-  Daryn is actively working on HADOOP-9747 for a while, and the patch are 
getting close to be committed. However, according to Daryn, the patch seems to 
cause some regression in some corner cases in secured environment (Storm auto 
tgt, etc.). May need some additional watch/review on this JIRA's fixes.



My monitoring JACC report between 2.8.2 and 2.7.4 will get finish tomorrow. If 
everything is going smoothly, I am planning to kick off RC0 around holiday 
(this weekend).



Thanks,



Junping




From: Brahma Reddy Battula 
Sent: Tuesday, August 29, 2017 8:42 AM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Hi All

Update on 2.8.2 release status
we are down to 3 critical issues ( YARN-6091,YARN-7083,HADOOP-9747),all are 
patch available and closer to commit.
Junping is closing tracking this.

Todo:

1) Update pom.xml ..?  currently it's with 2.8.3
https://github.com/apache/hadoop/blob/branch-2.8.2/pom.xml#L21
2) Wiki is 
outdated, need to update the wiki..?
3) As this is going to stable release,are we planing enable Big top for 2.8.2 
testing Or Dynamometer testing (anybody from linked-in can help)..?

@Junping Du,Please correct me,if I am wrong.


--Brahma Reddy Battula

From: Junping Du 
Sent: Monday, August 7, 2017 2:44 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Hello community,
Here is a quick update on status for 2.8.2:
- We are 0 blockers now!
- Still 9 critical issues, 8 of them are Patch Available and with actively 
working.
For details of pending blocker/critical issues, please refer: 
https://s.apache.org/JM5x
Issue Navigator - ASF JIRA
s.apache.org
Linked Applications. Loading… Dashboards




I am planning to cut off first RC in week of Aug. 21st to give these 
critical issues a bit more time (~2 weeks) to get fixed. Let's working towards 
first production GA release of Apache Hadoop 2.8 - let me know if you have any 
thoughts or comments.

Cheers,

Junping

From: Junping Du 
Sent: Monday, July 24, 2017 1:41 PM
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re:

I have done the change.

All committers,

  2.8.2 release is supposed to be a stable/production release for 
branch-2.8. For commits to go for 2.8.2 release (only important and low risk 
bug fixes), please commit to trunk, branch-2, branch-2.8 and branch-2.8.2. For 
unimportant or high risk bug fixes/improvements, please commit to branch-2.8 
(trunk/branch-2) only and mark JIRA fixed as 2.8.3. Thanks for your cooperation!


Thanks,


Junping



From: Junping Du
Sent: Monday, July 24, 2017 10:36 AM
To: Brahma Reddy Battula; Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Jason Lowe; 
humbed...@apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan


Nice catch, Brahma. Actually, this is not supposed to be happen as we all 
should know the patch should firstly get landed on branch-2.8 (as well as 
trunk, branch-2) before landed on branch-2.8.2. Anyway, we will always check 
JIRAs claim to fixed in a specific release version with commits actually 
landing on the releasing branch before kicking off RC. So, I am not too worry 
about this mistaking behaviors as it happens in every releases.

If

Re: [DISCUSS] Branches and versions for Hadoop 3

2017-08-30 Thread varunsax...@apache.org

Hi Andrew,

We have completed the merge of TSv2 to trunk.
You can now go ahead with the branching.

Regards,
Varun Saxena.

On Tue, Aug 29, 2017 at 11:35 PM, Andrew Wang 
wrote:

> Sure. Ping me when the TSv2 goes in, and I can take care of branching.
>
> We're still waiting on the native services and S3Guard merges, but I don't
> want to hold branching to the last minute.
>
> On Tue, Aug 29, 2017 at 10:51 AM, Vrushali C 
> wrote:
>
>> Hi Andrew,
>> As Rohith mentioned, if you are good with it, from the TSv2 side, we are
>> ready to go for merge tonight itself (Pacific time)  right after the voting
>> period ends. Varun Saxena has been diligently rebasing up until now so most
>> likely our merge should be reasonably straightforward.
>>
>> @Wangda: your resource profile vote ends tomorrow, could we please
>> coordinate our merges?
>>
>> thanks
>> Vrushali
>>
>>
>> On Mon, Aug 28, 2017 at 10:45 PM, Rohith Sharma K S <
>> rohithsharm...@apache.org> wrote:
>>
>>> On 29 August 2017 at 06:24, Andrew Wang 
>>> wrote:
>>>
>>> > So far I've seen no -1's to the branching proposal, so I plan to
>>> execute
>>> > this tomorrow unless there's further feedback.
>>> >
>>> For on going branch merge threads i.e TSv2, voting will be closing
>>> tomorrow. Does it end up in merging into trunk(3.1.0-SNAPSHOT) and
>>> branch-3.0(3.0.0-beta1-SNAPSHOT) ? If so, would you be able to wait for
>>> couple of more days before creating branch-3.0 so that TSv2 branch merge
>>> would be done directly to trunk?
>>>
>>>
>>>
>>> >
>>> > Regarding the above discussion, I think Jason and I have essentially
>>> the
>>> > same opinion.
>>> >
>>> > I hope that keeping trunk a release branch means a higher bar for
>>> merges
>>> > and code review in general. In the past, I've seen some patches
>>> committed
>>> > to trunk-only as a way of passing responsibility to a future user or
>>> > reviewer. That doesn't help anyone; patches should be committed with
>>> the
>>> > intent of running them in production.
>>> >
>>> > I'd also like to repeat the above thanks to the many, many contributors
>>> > who've helped with release improvements. Allen's work on
>>> create-release and
>>> > automated changes and release notes were essential, as was Xiao's work
>>> on
>>> > LICENSE and NOTICE files. I'm also looking forward to Marton's site
>>> > improvements, which addresses one of the remaining sore spots in the
>>> > release process.
>>> >
>>> > Things have gotten smoother with each alpha we've done over the last
>>> year,
>>> > and it's a testament to everyone's work that we have a good
>>> probability of
>>> > shipping beta and GA later this year.
>>> >
>>> > Cheers,
>>> > Andrew
>>> >
>>> >
>>>
>>
>>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-30 Thread Vrushali C

I'm adding my +1 (binding) to conclude the vote.

With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
the merge to trunk shortly. Thanks everyone!

Regards
Vrushali


On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
varun.saxena.apa...@gmail.com> wrote:

> +1 (binding).
>
> Kudos to all the team members for their great work!
>
> Being part of the ATSv2 team, I have been involved with either development
> or review of most of the JIRAs'.
> Tested ATSv2 in both secure and non-secure mode. Also verified that there
> is no impact when ATSv2 is turned off.
>
> Regards,
> Varun Saxena.
>
> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> vrushalic2...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> will
>> run for 7 days, and will end August 29 11:00 PM PDT.
>>
>> We have previously completed one merge onto trunk [3] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> confident that the state of the feature meets the criteria to be merged
>> onto trunk and we'd love folks to get their hands on it in a test capacity
>> and provide valuable feedback so that we can make it production-ready.
>>
>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>> usability improvements based on a new architecture. What we would like to
>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> complete end-to-end read/write flow with security and read level
>> authorization via whitelists. You should be able to start setting it up
>> and
>> testing it.
>>
>> At a high level, the following are the key features that have been
>> implemented since alpha1:
>> - Security via Kerberos Authentication and delegation tokens
>> - Read side simple authorization via whitelist
>> - Client configurable entity sort ordering
>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>> timerange, pagination, sub-app entities
>> - Support for storing sub-application entities (entities that exist
>> outside
>> the scope of an application)
>> - Configurable TTLs (time-to-live) for tables, configurable table
>> prefixes,
>> configurable hbase cluster
>> - Flow level aggregations done as dynamic (table level) coprocessors
>> - Uses latest stable HBase release 1.2.6
>>
>> There are a total of 82 subtasks that were completed as part of this
>> effort.
>>
>> We paid close attention to ensure that once disabled Timeline Service v.2
>> does not impact existing functionality when disabled (by default).
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Regards,
>> Vrushali
>>
>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>
>
>

[jira] [Created] (YARN-7139) FairScheduler: finished applications are always restored to default queue

[jira] [Created] (YARN-7138) Fix incompatible API change for YarnScheduler involved by YARN-5521

[jira] [Created] (YARN-7137) Resource profile: Move new added APIs to unstable

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[jira] [Created] (YARN-7136) Additional Performance Improvement for Resource Profile Feature

[jira] [Created] (YARN-7135) Clean up lock-try order in common scheduler code

[jira] [Created] (YARN-7134) AppSchedulingInfo has a dependency on capacity scheduler

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[jira] [Created] (YARN-7133) Clean up lock-try order fair scheduler

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[jira] [Created] (YARN-7132) FairScheduler.initScheduler() contains a surprising unary plus

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[jira] [Created] (YARN-7131) FSDownload.unpack should read determine the type of resource by reading the header bytes

[jira] [Created] (YARN-7130) ATSv2 documentation changes post merge

[jira] [Created] (YARN-7129) Application Catalog for YARN applications

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[jira] [Resolved] (YARN-7111) ApplicationHistoryServer webpage startTime and state are not readable

[jira] [Created] (YARN-7128) The error message in TimelineSchemaCreator is not enough to find out the error.

Re: Apache Hadoop 2.8.2 Release Plan

Re: [DISCUSS] Branches and versions for Hadoop 3

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

21 matches

Site Navigation

Mail list logo

Footer information