date:20160812

[jira] [Created] (YARN-5520) [Capacity Scheduler] Change the logic for when to trigger user/group mappings to queues

2016-08-12 Thread Min Shen (JIRA)

Min Shen created YARN-5520:
--

 Summary: [Capacity Scheduler] Change the logic for when to trigger 
user/group mappings to queues
 Key: YARN-5520
 URL: https://issues.apache.org/jira/browse/YARN-5520
 Project: Hadoop YARN
  Issue Type: Improvement
Affects Versions: 2.6.1, 2.7.0, 2.6.0
Reporter: Min Shen


In YARN-2411, the feature in Capacity Scheduler to support user/group based 
mappings to queues was introduced.
In the original implementation, the configuration key 
{{yarn.scheduler.capacity.queue-mappings-override.enable}} was added to control 
when to enable overriding user requested queues.
However, even if this configuration is set to false, queue overriding could 
still happen if the user didn't request for any specific queue or choose to 
simply submit his job to "default" queue, according to the following if 
condition which triggers queue overriding:
{code}
if (queueName.equals(YarnConfiguration.DEFAULT_QUEUE_NAME)
  || overrideWithQueueMappings)
{code}

This logic does not seem very reasonable, as there's no way to fully disable 
queue overriding when mappings are configured inside capacity-scheduler.xml.

In addition, in our environment, we have setup a few organization dedicated 
queues as well as some "adhoc" queues. The organization dedicated queues have 
better resource guarantees and we want to be able to route users to the 
corresponding organization queues. On the other hand, the "adhoc" queues have 
less resource guarantees but everyone can use it to get some opportunistic 
resources when the cluster is free.
The current logic will also prevent this type of use cases as when you enable 
queue overriding, users cannot use these "adhoc" queues any more. They will 
always be routed to the dedicated organization queues.

To address the above 2 issues, I propose to change the implementation so that:
* Admin can fully disable queue overriding even if mappings are already 
configured.
* Admin have finer grained control to cope queue overriding with the above 
mentioned organization/adhoc queue setups.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5519) Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover

2016-08-12 Thread Giovanni Matteo Fumarola (JIRA)

Giovanni Matteo Fumarola created YARN-5519:
--

 Summary: Add SubClusterId in AddApplicationHomeSubClusterResponse 
for Router Failover
 Key: YARN-5519
 URL: https://issues.apache.org/jira/browse/YARN-5519
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Giovanni Matteo Fumarola
Assignee: Ellen Hui


This JIRA tracks the addition of SubClusterId into 
AddApplicationHomeSubClusterResponse. 
in the design of [YARN-3659|https://issues.apache.org/jira/browse/YARN-3659], 
to handle better fail-over scenario the response needs SubclusterId as field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5518) add HTTP_AND_HTTPS policy support in WebApps

2016-08-12 Thread Eric Feng (JIRA)

Eric Feng created YARN-5518:
---

 Summary: add HTTP_AND_HTTPS policy support in WebApps
 Key: YARN-5518
 URL: https://issues.apache.org/jira/browse/YARN-5518
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: api, resourcemanager, webapp
Reporter: Eric Feng


Currently WebApps/WebServers in Hadoop only support HTTP_ONLY and HTTPS_ONLY 
policy, which means there will be a service interruption when we change the 
policy from HTTP_ONLY to HTTPS_ONLY.

The root cause of this limit is that org.apache.hadoop.yarn.webapp.WebApps 
supports only one endpoint, but the http server(HttpServer2) behind it do 
support multiple endpoints. So if we add multiple endpoints support in WebApps, 
we can add both HTTP endpoint and HTTPS endpoint in all web servers in Hadoop, 
i.e. support HTTP_AND_HTTPS policy. with this support, we will be able to 
switch the web protocol from HTTP to HTTPS by 2 config changes without service 
interruption:
1. change policy from HTTP_ONLY to HTTP_AND_HTTPS;
2. change policy form HTTP_AND_HTTPS to HTTPS_ONLY;

This JIRA tracks changes to WebApps and web services/clients to support 
HTTP_AND_HTTPS policy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[DISCUSS] Release cadence and EOL

2016-08-12 Thread Karthik Kambatla

Forking off this discussion from 2.6.5 release thread. Junping and Chris T
have brought up important concerns regarding too many concurrent releases
and the lack of EOL for our releases.

First up, it would be nice to hear from others on our releases having the
notion of EOL and other predictability is indeed of interest.

Secondly, I believe EOLs work better in conjunction with a predictable
cadence. Given past discussions on this and current periods between our
minor releases, I would like to propose a minor release on the latest major
line every 6 months and a maintenance release on the latest minor release
every 2 months.

Eager to hear others thoughts.

Thanks
Karthik

Re: [Release thread] 2.6.5 release activities

2016-08-12 Thread Sangjin Lee

Thanks folks for opening up the discussion on our EOL policy. That's also
exactly what I wanted to discuss when I opened up the 2.6.5 discussion:

I also want to gauge the community's interest in maintaining the 2.6.x
> line. How long do we maintain this line? What would be a sensible EOL
> policy? Note that as the main code lines start diverging (java version,
> features, etc.), the cost of maintaining multiple release lines does go up.
> I'd love to hear your thoughts.


I look forward to that discussion thread. As for 2.6.5, since there is
enough interest in it, I'll work with Chris on that.

Regards,
Sangjin

On Fri, Aug 12, 2016 at 8:19 AM, Junping Du  wrote:

> I am OK with going forward for 2.6.5 release given some people may not be
> ready for migration to 2.7 and we have done some preparation work already.
> Thanks Chris for pushing the release work forward!
> About future releases of 2.6 (after 2.6.5) or any other legacy releases, I
> think it worth more discussions. I disagree with what Allen said below -
> Java 6 support should be a solid reason for branch-2.6 release keep going.
> In this community, we are so aggressive to drop Java 7 support in 3.0.x
> release. Here, why we are so conservative to keep releasing new bits to
> support Java 6? Our release effort, although driven by different person,
> should be consistent logically.
> I don't think we have clear rules/polices about EOL of legacy releases
> before. This is a bit off topic here. Will start a new thread later for
> more discussion.
>
> Thanks,
>
> Junping
>
> 
> From: Chris Trezzo 
> Sent: Friday, August 12, 2016 12:42 AM
> To: Karthik Kambatla
> Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: Re: [Release thread] 2.6.5 release activities
>
> Thanks everyone for the discussion! Based on what has been said in this
> thread, I will move forward on the preparation efforts (working with
> Sangjin and the community) for a 2.6.5 release. If there is a strong
> objection to this, please let us know.
>
> I see three initial tasks going forward:
>
> 1. Fix the pre-commit build for branch-2.6. I am not entirely sure what is
> involved here, but will start looking into it using HADOOP-12800 as a
> starting point.
>
> 2. Look through the unresolved issues still targeted to 2.6.5 and
> resolve/re-target as necessary. There are currently 17 of them:
> https://issues.apache.org/jira/issues/?jql=(project%20%
> 3D%20%22Hadoop%20Common%22%20OR%20project%20%3D%20%
> 22Hadoop%20YARN%22%20OR%
> 20project%20%3D%20%22Hadoop%20Map%2FReduce%22%20OR%
> 20project%20%3D%20%22Hadoop%20HDFS%22)%20AND%20(
> fixVersion%20%3D%202.6.5%20OR%20%22Target%20Version%2Fs%22%
> 20%3D%202.6.5)%20AND%20(status%20!%3D%20Resolved%20AND%20status%20!%3D%
> 20Closed)%20ORDER%20BY%20status%20ASC
>
> 3. Look through the backport candidates in the spreadsheet in more detail
> and backport/drop as necessary. There are currently 15 of them:
> https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC
> 8hTRUemHvYPPzY/edit?usp=sharing
>
> Finally, I think it would be awesome to continue the release end-of-life
> discussion. As we get better and better at release cadence it is going to
> be really helpful to have an established EOL policy. It will be less
> confusing for contributors and help set clear expectations for end users as
> to when they need to start making reasonable migration plans, especially if
> they want to stay on a well supported release line. I would be happy to
> help with this effort as well. It would be great if we could leverage that
> discussion and have EOL information for the 2.6 line when we release 2.6.5.
>
> As always, please let me know if I have missed something.
>
> Thanks!
> Chris Trezzo
>
> On Thu, Aug 11, 2016 at 1:03 PM, Karthik Kambatla 
> wrote:
>
> > Since there is sufficient interest in 2.6.5, we should probably do it.
> All
> > the reasons Allen outlines make sense.
> >
> > That said, Junping brings up a very important point that we should think
> of
> > for future releases. For a new user or a user that does not directly
> > contribute to the project, more stable releases make it hard to pick
> from.
> >
> > As Chris T mentioned, the notion of EOL for our releases seems like a
> good
> > idea. However, to come up with any EOLs, we need to have some sort of
> > cadence for the releases. While this is hard for major releases (big
> bang,
> > potentially incompatible features), it should be doable for minor
> releases.
> >
> > How do people feel about doing a minor release every 6 months, with
> > follow-up maintenance releases every 2 months until the next minor and as
> > needed after that? That way, we could EOL a minor release a year after
> its
> > initial release? In the future, we could consider shrinking this window.
> In
> > addition to the EOL, this also makes our releases a little more
> predictable
>

Re: [Release thread] 2.6.5 release activities

2016-08-12 Thread Kihwal Lee

You might want the snapshot bug fix done in HDFS-7056. This bug creates 
snapshot filediffs even if you never use snapshot. For 2.6, we will have to do 
it in a separate jira to pick up the fix only. Related to this, HDFS-9696 might 
be of interest too.
Kihwal

  From: Chris Trezzo 
 To: Karthik Kambatla  
Cc: "common-...@hadoop.apache.org" ; 
"hdfs-...@hadoop.apache.org" ; 
"mapreduce-...@hadoop.apache.org" ; 
"yarn-dev@hadoop.apache.org" 
 Sent: Thursday, August 11, 2016 6:42 PM
 Subject: Re: [Release thread] 2.6.5 release activities

Thanks everyone for the discussion! Based on what has been said in this
thread, I will move forward on the preparation efforts (working with
Sangjin and the community) for a 2.6.5 release. If there is a strong
objection to this, please let us know.

I see three initial tasks going forward:

1. Fix the pre-commit build for branch-2.6. I am not entirely sure what is
involved here, but will start looking into it using HADOOP-12800 as a
starting point.

2. Look through the unresolved issues still targeted to 2.6.5 and
resolve/re-target as necessary. There are currently 17 of them:
https://issues.apache.org/jira/issues/?jql=(project%20%
3D%20%22Hadoop%20Common%22%20OR%20project%20%3D%20%22Hadoop%20YARN%22%20OR%
20project%20%3D%20%22Hadoop%20Map%2FReduce%22%20OR%
20project%20%3D%20%22Hadoop%20HDFS%22)%20AND%20(
fixVersion%20%3D%202.6.5%20OR%20%22Target%20Version%2Fs%22%
20%3D%202.6.5)%20AND%20(status%20!%3D%20Resolved%20AND%20status%20!%3D%
20Closed)%20ORDER%20BY%20status%20ASC

3. Look through the backport candidates in the spreadsheet in more detail
and backport/drop as necessary. There are currently 15 of them:
https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC
8hTRUemHvYPPzY/edit?usp=sharing

Finally, I think it would be awesome to continue the release end-of-life
discussion. As we get better and better at release cadence it is going to
be really helpful to have an established EOL policy. It will be less
confusing for contributors and help set clear expectations for end users as
to when they need to start making reasonable migration plans, especially if
they want to stay on a well supported release line. I would be happy to
help with this effort as well. It would be great if we could leverage that
discussion and have EOL information for the 2.6 line when we release 2.6.5.

As always, please let me know if I have missed something.

Thanks!
Chris Trezzo

On Thu, Aug 11, 2016 at 1:03 PM, Karthik Kambatla 
wrote:

> Since there is sufficient interest in 2.6.5, we should probably do it. All
> the reasons Allen outlines make sense.
>
> That said, Junping brings up a very important point that we should think of
> for future releases. For a new user or a user that does not directly
> contribute to the project, more stable releases make it hard to pick from.
>
> As Chris T mentioned, the notion of EOL for our releases seems like a good
> idea. However, to come up with any EOLs, we need to have some sort of
> cadence for the releases. While this is hard for major releases (big bang,
> potentially incompatible features), it should be doable for minor releases.
>
> How do people feel about doing a minor release every 6 months, with
> follow-up maintenance releases every 2 months until the next minor and as
> needed after that? That way, we could EOL a minor release a year after its
> initial release? In the future, we could consider shrinking this window. In
> addition to the EOL, this also makes our releases a little more predictable
> for both users and vendors. Note that 2.6.0 went out almost 2 years ago and
> we haven't had a new minor release in 14 months. I am happy to start
> another DISCUSS thread around this if people think it is useful.
>
> Thanks
> Karthik
>
> On Thu, Aug 11, 2016 at 12:50 PM, Allen Wittenauer <
> a...@effectivemachines.com
> > wrote:
>
> >
> > > On Aug 11, 2016, at 8:10 AM, Junping Du  wrote:
> > >
> > > Allen, to be clear, I am not against any branch release effort here.
> > However,
> >
> >                "I'm not an X but "
> >
> > > as RM for previous releases 2.6.3 and 2.6.4, I feel to have
> > responsibility to take care branch-2.6 together with other RMs (Vinod and
> > Sangjin) on this branch and understand current gap - especially, to get
> > consensus from community on the future plan for 2.6.x.
> > > Our bylaw give us freedom for anyone to do release effort, but our
> bylaw
> > doesn't stop our rights for reasonable question/concern on any release
> > plan. As you mentioned below, people can potentially fire up branch-1
> > release effort. But if you call a release plan tomorrow for branch-1, I
> > cannot imagine nobody will question on that effort. Isn't it?
> >
> >                From previous discussions I've seen around releases, I
> > think it would depend upon which employee from which vendor raised the
> > question.
> >
> > > Let's keep discussions on releasing 2.6.5 more technical. IMO, to make
> > 2.6.5 release

[VOTE] Release Apache Hadoop 2.7.3 RC1

2016-08-12 Thread Vinod Kumar Vavilapalli

Hi all,

I've created a release candidate RC1 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 


The RC tag in git is: release-2.7.3-RC1

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2]: 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html

Re: [Release thread] 2.6.5 release activities

2016-08-12 Thread Junping Du

I am OK with going forward for 2.6.5 release given some people may not be ready 
for migration to 2.7 and we have done some preparation work already. Thanks 
Chris for pushing the release work forward!
About future releases of 2.6 (after 2.6.5) or any other legacy releases, I 
think it worth more discussions. I disagree with what Allen said below - Java 6 
support should be a solid reason for branch-2.6 release keep going. In this 
community, we are so aggressive to drop Java 7 support in 3.0.x release. Here, 
why we are so conservative to keep releasing new bits to support Java 6? Our 
release effort, although driven by different person, should be consistent 
logically. 
I don't think we have clear rules/polices about EOL of legacy releases before. 
This is a bit off topic here. Will start a new thread later for more discussion.

Thanks,

Junping

From: Chris Trezzo 
Sent: Friday, August 12, 2016 12:42 AM
To: Karthik Kambatla
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [Release thread] 2.6.5 release activities

Thanks everyone for the discussion! Based on what has been said in this
thread, I will move forward on the preparation efforts (working with
Sangjin and the community) for a 2.6.5 release. If there is a strong
objection to this, please let us know.

I see three initial tasks going forward:

1. Fix the pre-commit build for branch-2.6. I am not entirely sure what is
involved here, but will start looking into it using HADOOP-12800 as a
starting point.

2. Look through the unresolved issues still targeted to 2.6.5 and
resolve/re-target as necessary. There are currently 17 of them:
https://issues.apache.org/jira/issues/?jql=(project%20%
3D%20%22Hadoop%20Common%22%20OR%20project%20%3D%20%22Hadoop%20YARN%22%20OR%
20project%20%3D%20%22Hadoop%20Map%2FReduce%22%20OR%
20project%20%3D%20%22Hadoop%20HDFS%22)%20AND%20(
fixVersion%20%3D%202.6.5%20OR%20%22Target%20Version%2Fs%22%
20%3D%202.6.5)%20AND%20(status%20!%3D%20Resolved%20AND%20status%20!%3D%
20Closed)%20ORDER%20BY%20status%20ASC

3. Look through the backport candidates in the spreadsheet in more detail
and backport/drop as necessary. There are currently 15 of them:
https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC
8hTRUemHvYPPzY/edit?usp=sharing

Finally, I think it would be awesome to continue the release end-of-life
discussion. As we get better and better at release cadence it is going to
be really helpful to have an established EOL policy. It will be less
confusing for contributors and help set clear expectations for end users as
to when they need to start making reasonable migration plans, especially if
they want to stay on a well supported release line. I would be happy to
help with this effort as well. It would be great if we could leverage that
discussion and have EOL information for the 2.6 line when we release 2.6.5.

As always, please let me know if I have missed something.

Thanks!
Chris Trezzo

On Thu, Aug 11, 2016 at 1:03 PM, Karthik Kambatla 
wrote:

> Since there is sufficient interest in 2.6.5, we should probably do it. All
> the reasons Allen outlines make sense.
>
> That said, Junping brings up a very important point that we should think of
> for future releases. For a new user or a user that does not directly
> contribute to the project, more stable releases make it hard to pick from.
>
> As Chris T mentioned, the notion of EOL for our releases seems like a good
> idea. However, to come up with any EOLs, we need to have some sort of
> cadence for the releases. While this is hard for major releases (big bang,
> potentially incompatible features), it should be doable for minor releases.
>
> How do people feel about doing a minor release every 6 months, with
> follow-up maintenance releases every 2 months until the next minor and as
> needed after that? That way, we could EOL a minor release a year after its
> initial release? In the future, we could consider shrinking this window. In
> addition to the EOL, this also makes our releases a little more predictable
> for both users and vendors. Note that 2.6.0 went out almost 2 years ago and
> we haven't had a new minor release in 14 months. I am happy to start
> another DISCUSS thread around this if people think it is useful.
>
> Thanks
> Karthik
>
> On Thu, Aug 11, 2016 at 12:50 PM, Allen Wittenauer <
> a...@effectivemachines.com
> > wrote:
>
> >
> > > On Aug 11, 2016, at 8:10 AM, Junping Du  wrote:
> > >
> > > Allen, to be clear, I am not against any branch release effort here.
> > However,
> >
> > "I'm not an X but "
> >
> > > as RM for previous releases 2.6.3 and 2.6.4, I feel to have
> > responsibility to take care branch-2.6 together with other RMs (Vinod and
> > Sangjin) on this branch and understand current gap - especially, to get
> > consensus from community on the future plan for 2.6.x.
> > > Our bylaw give us f

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2016-08-12 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/

[Aug 12, 2016 7:05:52 AM] (kai.zheng) HADOOP-11588. Benchmark framework and 
test for erasure coders.
[Aug 11, 2016 6:57:43 PM] (weichiu) HADOOP-13441. Document LdapGroupsMapping 
keystore password properties.
[Aug 11, 2016 7:27:09 PM] (weichiu) HADOOP-13190. Mention 
LoadBalancingKMSClientProvider in KMS HA
[Aug 11, 2016 7:39:41 PM] (naganarasimha_gr) YARN-4833. For Queue 
AccessControlException client retries multiple
[Aug 12, 2016 2:56:58 AM] (sjlee) HADOOP-13410. RunJar adds the content of the 
jar twice to the classpath
[Aug 13, 2016 5:52:37 AM] (kai.zheng) HDFS-8668. Erasure Coding: revisit buffer 
used for encoding and




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.yarn.logaggregation.TestAggregatedLogFormat 
   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.api.impl.TestYarnClient 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/diff-compile-javac-root.txt
  [172K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/diff-patch-pylint.txt
  [16K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
  [24K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [124K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/131/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Short peaks in container memory usage

2016-08-12 Thread Ravi Prakash

Hi Jan!

Yes! Makes sense. I'm sure there were bigger changes for the
ResourceHandler. Which version are you on?

Cheers
Ravi

On Thu, Aug 11, 2016 at 7:48 AM, Jan Lukavský 
wrote:

> Hi Ravi,
>
> I don't think cgroups will help us, because, we don't want to impose a
> hard limit on the memory usage, we just want to allow for short time
> periods, when container can consume more memory than its limit. We don't
> want to put the limit too high, because that causes underutilization of our
> cluster, but setting it "reasonable" causes applications to fail (because
> of random containers being killed because of spikes). That's why we created
> the time-window averaging resource calculator, and I was trying to find
> out, if anybody else is having similar kind of issues. If so, I could
> contribute our extension (and therefore we will not have to maintain it
> ourselves in a separate repository :)). The resource calculator is for
> hadoop 2.6, and I suppose there might be larger changes around this in
> higher versions?
>
> Cheers,
>  Jan
>
> On 10.8.2016 19:23, Ravi Prakash wrote:
>
> Hi Jan!
>
> Thanks for your explanation. I'm glad that works for you! :-)
> https://issues.apache.org/jira/browse/YARN-5202 is something that Yahoo!
> talked about at the Hadoop Summit, (and it seems the community may be going
> in a similar direction, although not exactly the same.) There's also
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-
> project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-
> server-nodemanager/src/main/java/org/apache/hadoop/yarn/
> server/nodemanager/containermanager/linux/resources/CGroupsHandler.java .
> Ideally at my company we'd like memory limits also to be imposed by Cgroups
> because we have had the OOM-killer wreak havoc a couple of times, but from
> what I know, that is not an option yet.
>
> Cheers
> Ravi
>
> On Wed, Aug 10, 2016 at 1:54 AM, Jan Lukavský <
> jan.lukav...@firma.seznam.cz> wrote:
>
>> Hi Ravi,
>>
>> we don't run into situation where memory used > RAM, because memory
>> configured to be used by all containers on a node is less than the total
>> amount on memory (by a factor of say 10%). The spikes of container memory
>> usage, that are tolerated due to the averaging don't happen on all
>> containers at once, but are more of a random nature and therefore mostly
>> only single running container "spikes", which therefore doesn't cause any
>> issues. To fully answer your question, we have overcommit enabled and
>> therefore, if we would run out of memory, bad things would happen. :) We
>> are aware of that. The risk of running into OOM-killer can be controlled by
>> the averaging window length - as the length grows, the more and more spikes
>> are tolerated. Setting the averaging window length to 1 switches this
>> feature off, turning it back into the "standard" behavior, which is why I
>> see it as a extension of the current approach, which could be interesting
>> to other people as well.
>>
>>   Jan
>>
>>
>> On 10.8.2016 02:48, Ravi Prakash wrote:
>>
>> Hi Jan!
>>
>> Thanks for your contribution. In your approach what happens when a few
>> containers on a node are using "excessive" memory (so that total memory
>> used > RAM available on the machine). Do you have overcommit enabled?
>>
>> Thanks
>> Ravi
>>
>> On Tue, Aug 9, 2016 at 1:31 AM, Jan Lukavský <
>> jan.lukav...@firma.seznam.cz> wrote:
>>
>>> Hello community,
>>>
>>> I have a question about container resource calculation in nodemanager.
>>> Some time ago a filed JIRA https://issues.apache.org/jira
>>> /browse/YARN-4681, which I though might address our problems with
>>> container being killed because of read-only mmaping memory block. The JIRA
>>> has not been resolved yet, but it turned out for us, that the patch doesn't
>>> solve the problem. Some applications (namely Apache Spark) tend to allocate
>>> really large memory blocks outside JVM heap (using mmap, but with
>>> MAP_PRIVATE), but only for short time periods. We solved this by creating a
>>> smoothing resource calculator, which averages the memory usage of a
>>> container over some time period (say 5 minutes). This eliminates the
>>> problem of container being killed for short memory consumption peak, but in
>>> the same time preserves the ability to kill container that *really*
>>> consumes excessive amount of memory.
>>>
>>> My question is, does this seem a systematic approach to you and should I
>>> post our patch to the community or am thinking in a wrong direction from
>>> the beginning? :)
>>>
>>>
>>> Thanks for reactions,
>>>
>>>  Jan
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>>
>>>
>>
>>
>
>
> --
>
> Jan Lukavský
> Vedoucí týmu vývoje
> Seznam.cz, a.s.
> Radlická 3294/10
> 15000, Praha 5
> jan.lukavsky@firma.seznam.czhttp://www.seznam.cz
>
>

Re: [Release thread] 2.6.5 release activities

2016-08-12 Thread Chris Trezzo

Thanks everyone for the discussion! Based on what has been said in this
thread, I will move forward on the preparation efforts (working with
Sangjin and the community) for a 2.6.5 release. If there is a strong
objection to this, please let us know.

I see three initial tasks going forward:

1. Fix the pre-commit build for branch-2.6. I am not entirely sure what is
involved here, but will start looking into it using HADOOP-12800 as a
starting point.

2. Look through the unresolved issues still targeted to 2.6.5 and
resolve/re-target as necessary. There are currently 17 of them:
https://issues.apache.org/jira/issues/?jql=(project%20%
3D%20%22Hadoop%20Common%22%20OR%20project%20%3D%20%22Hadoop%20YARN%22%20OR%
20project%20%3D%20%22Hadoop%20Map%2FReduce%22%20OR%
20project%20%3D%20%22Hadoop%20HDFS%22)%20AND%20(
fixVersion%20%3D%202.6.5%20OR%20%22Target%20Version%2Fs%22%
20%3D%202.6.5)%20AND%20(status%20!%3D%20Resolved%20AND%20status%20!%3D%
20Closed)%20ORDER%20BY%20status%20ASC

3. Look through the backport candidates in the spreadsheet in more detail
and backport/drop as necessary. There are currently 15 of them:
https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC
8hTRUemHvYPPzY/edit?usp=sharing

Finally, I think it would be awesome to continue the release end-of-life
discussion. As we get better and better at release cadence it is going to
be really helpful to have an established EOL policy. It will be less
confusing for contributors and help set clear expectations for end users as
to when they need to start making reasonable migration plans, especially if
they want to stay on a well supported release line. I would be happy to
help with this effort as well. It would be great if we could leverage that
discussion and have EOL information for the 2.6 line when we release 2.6.5.

As always, please let me know if I have missed something.

Thanks!
Chris Trezzo

On Thu, Aug 11, 2016 at 1:03 PM, Karthik Kambatla 
wrote:

> Since there is sufficient interest in 2.6.5, we should probably do it. All
> the reasons Allen outlines make sense.
>
> That said, Junping brings up a very important point that we should think of
> for future releases. For a new user or a user that does not directly
> contribute to the project, more stable releases make it hard to pick from.
>
> As Chris T mentioned, the notion of EOL for our releases seems like a good
> idea. However, to come up with any EOLs, we need to have some sort of
> cadence for the releases. While this is hard for major releases (big bang,
> potentially incompatible features), it should be doable for minor releases.
>
> How do people feel about doing a minor release every 6 months, with
> follow-up maintenance releases every 2 months until the next minor and as
> needed after that? That way, we could EOL a minor release a year after its
> initial release? In the future, we could consider shrinking this window. In
> addition to the EOL, this also makes our releases a little more predictable
> for both users and vendors. Note that 2.6.0 went out almost 2 years ago and
> we haven't had a new minor release in 14 months. I am happy to start
> another DISCUSS thread around this if people think it is useful.
>
> Thanks
> Karthik
>
> On Thu, Aug 11, 2016 at 12:50 PM, Allen Wittenauer <
> a...@effectivemachines.com
> > wrote:
>
> >
> > > On Aug 11, 2016, at 8:10 AM, Junping Du  wrote:
> > >
> > > Allen, to be clear, I am not against any branch release effort here.
> > However,
> >
> > "I'm not an X but "
> >
> > > as RM for previous releases 2.6.3 and 2.6.4, I feel to have
> > responsibility to take care branch-2.6 together with other RMs (Vinod and
> > Sangjin) on this branch and understand current gap - especially, to get
> > consensus from community on the future plan for 2.6.x.
> > > Our bylaw give us freedom for anyone to do release effort, but our
> bylaw
> > doesn't stop our rights for reasonable question/concern on any release
> > plan. As you mentioned below, people can potentially fire up branch-1
> > release effort. But if you call a release plan tomorrow for branch-1, I
> > cannot imagine nobody will question on that effort. Isn't it?
> >
> > From previous discussions I've seen around releases, I
> > think it would depend upon which employee from which vendor raised the
> > question.
> >
> > > Let's keep discussions on releasing 2.6.5 more technical. IMO, to make
> > 2.6.5 release more reasonable, shouldn't we check following questions
> first?
> > > 1. Do we have any significant issues that should land on 2.6.5
> comparing
> > with 2.6.4?
> > > 2. If so, any technical reasons (like: upgrade is not smoothly,
> > performance downgrade, incompatibility with downstream projects, etc.) to
> > stop our users to move from 2.6.4 to 2.7.2/2.7.3?
> > > I believe having good answer on these questions can make our release
> > plan more reasonable to the whole community. More thoughts?
> >
> > I think these questions are

[jira] [Resolved] (YARN-5245) [YARN-3368] Add nodes heatmap chart

2016-08-12 Thread Sunil G (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G resolved YARN-5245.
---
Resolution: Done

> [YARN-3368] Add nodes heatmap chart
> ---
>
> Key: YARN-5245
> URL: https://issues.apache.org/jira/browse/YARN-5245
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: Screen Shot 2016-06-13 at 3.01.01 PM.png, Screen Shot 
> 2016-06-13 at 3.01.16 PM.png, YARN-5245.1.patch
>
>
> It gonna be very helpful to add heat map chart to get overall cluster 
> resource usage of nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5517) Add GPU as a resource type for scheduling on branch-2.7.1

2016-08-12 Thread Jaeboo Jeong (JIRA)

Jaeboo Jeong created YARN-5517:
--

 Summary: Add GPU as a resource type for scheduling on branch-2.7.1
 Key: YARN-5517
 URL: https://issues.apache.org/jira/browse/YARN-5517
 Project: Hadoop YARN
  Issue Type: Improvement
Reporter: Jaeboo Jeong


Currently YARN only support scheduling based on memory and cpu.
There is the issue(YARN-3926) which proposed to extend the YARN resource model.
And there is the issue(YARN-4122) to add support for GPU as a resource  using 
docker.

But these issues didn’t release yet so I just added GPU resource type like 
memory and cpu.
I don’t consider GPU isolation like YARN-4122.

The properties for GPU resource type is similar to cpu core.

mapred-default.xml
mapreduce.map.gpu.cores (default 0)
mapreduce.reduce.gpu.cores  (default 0)
yarn.app.mapreduce.am.resource.gpu-cores (default 0)

yarn-default.xml
yarn.scheduler.minimum-allocation-gcores (default 0)
yarn.scheduler.maximum-allocation-gcores (default 8)
yarn.nodemanager.resource.gcores (default 0)

I attached the patch for branch-2.7.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5520) [Capacity Scheduler] Change the logic for when to trigger user/group mappings to queues

[jira] [Created] (YARN-5519) Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover

[jira] [Created] (YARN-5518) add HTTP_AND_HTTPS policy support in WebApps

[DISCUSS] Release cadence and EOL

Re: [Release thread] 2.6.5 release activities

Re: [Release thread] 2.6.5 release activities

[VOTE] Release Apache Hadoop 2.7.3 RC1

Re: [Release thread] 2.6.5 release activities

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

Re: Short peaks in container memory usage

Re: [Release thread] 2.6.5 release activities

[jira] [Resolved] (YARN-5245) [YARN-3368] Add nodes heatmap chart

[jira] [Created] (YARN-5517) Add GPU as a resource type for scheduling on branch-2.7.1

13 matches

Site Navigation

Mail list logo

Footer information