date:20171030

[jira] [Resolved] (YARN-6508) Support FPGA plugin

2017-10-30 Thread Zhankun Tang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhankun Tang resolved YARN-6508.

Resolution: Implemented

> Support FPGA plugin
> ---
>
> Key: YARN-6508
> URL: https://issues.apache.org/jira/browse/YARN-6508
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Zhankun Tang
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Hadoop Compatability Guide, Part Deux: Developer Docs

2017-10-30 Thread Daniel Templeton

We've now gone a couple of rounds of reviews on HADOOP-14876, and a 
patch is posted for HADOOP-14875.  Feedback is very welcome.  Please 
take a look.


Daniel

On 10/14/17 8:46 AM, Daniel Templeton wrote:
I just posted a first patch for HADOOP-14876 that adds downstream 
developer docs based on the overhauled compatibility guide from 
HADOOP-13714.  I would really appreciate some critical review of the 
doc, as it's much more likely to be read by downstream developers than 
the compatibility spec itself.


There's another doc coming in HADOOP-14875 that will add what amounts 
to upgrade docs for admins.  When that's complete, I will send another 
email here to solicit reviews.


Daniel



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Cutting branch-2.9

2017-10-30 Thread Junping Du

Hi Subru and Arun,
 Thanks for moving forward with 2.9 release. Is the first cut of 2.9 
release supposed to be a stable version or just an alpha version? If it is 
supposed to be a stable version, we should run jdiff test and check for API 
compatibility before releasing out. Please let me know if you need any help 
here.

Thanks,

Junping

From: Subru Krishnan 
Sent: Monday, October 30, 2017 12:39 PM
To: common-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Cc: Arun Suresh
Subject: Cutting branch-2.9

We want to give heads up that we are going to cut branch-2.9 tomorrow
morning.

We are down to 3 blockers and they all are close to being committed (thanks
everyone):
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release

There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which are
close to completion.

Folks who are working/reviewing these, kindly prioritize accordingly so
that we can make the release on time.
https://issues.apache.org/jira/browse/YARN-7398?filter=12342468

Thanks in advance!

-Subru/Arun



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7417) re-factory IndexedFileAggregatedLogsBlock and TFileAggregatedLogsBlock to remove duplicate codes

2017-10-30 Thread Xuan Gong (JIRA)

Xuan Gong created YARN-7417:
---

 Summary: re-factory IndexedFileAggregatedLogsBlock and 
TFileAggregatedLogsBlock to remove duplicate codes
 Key: YARN-7417
 URL: https://issues.apache.org/jira/browse/YARN-7417
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Xuan Gong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Merge yarn-native-services branch into trunk

2017-10-30 Thread Jian He

Few more things:

This is the document for trying a non-docker service on YARN.
https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/QuickStart.md

And the document for a docker based service
https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/Examples.md

And the vote lasts 7 days as usual.

Thanks,
Jian

On Oct 30, 2017, at 1:06 PM, Jian He
> wrote:

Hi All,

I would like to restart the vote for merging yarn-native-services to trunk.
Since last vote, we have been working on several issues in documentation, DNS,
CLI modifications etc. We believe now the feature is in a much better shape.

Some back ground:
At a high level, the following are the key feautres implemented.
- YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate
existing services to YARN either docker or non-docker based.
- YARN-4793[2]. A Rest API service embeded in RM (optional) for user to deploy
a service via a simple JSON spec
- YARN-4757[3]. Extending today's service registry with a simple DNS service to
enable users to discover services deployed on YARN via standard DNS lookup
- YARN-6419[4]. UI support for native-services on the new YARN UI
All these new services are optional and are sitting outside of the existing
system, and have no impact on existing system if disabled.

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi,
Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G,
Akhil PB, Eric Yang. This effort could not be possible without their ideas and
hard work.
Also thanks Allen for some review and verifications.

Thanks,
Jian

[1] https://issues.apache.org/jira/browse/YARN-5079
[2] https://issues.apache.org/jira/browse/YARN-4793
[3] https://issues.apache.org/jira/browse/YARN-4757
[4] https://issues.apache.org/jira/browse/YARN-6419

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-10-30 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/574/

[Oct 29, 2017 11:44:16 PM] (yufei) YARN-6747. 
TestFSAppStarvation.testPreemptionEnable fails
[Oct 30, 2017 1:54:33 AM] (templedf) YARN-7374. Improve performance of DRF 
comparisons for resource types in




-1 overall


The following subsystems voted -1:
asflicense findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-tools/hadoop-resourceestimator 
   Dead store to jobHistory in 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.getHistoryResourceSkyline(String,
 String) At 
ResourceEstimatorService.java:org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.getHistoryResourceSkyline(String,
 String) At ResourceEstimatorService.java:[line 196] 
   Incorrect lazy initialization and update of static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore
 in new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() 
At ResourceEstimatorService.java:of static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore
 in new org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() 
At ResourceEstimatorService.java:[lines 78-82] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.config 
from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 80] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.gson from 
instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 106] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.logParser 
from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 86] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.rleType 
from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 108] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStore
 from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 82] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.skylineStoreType
 from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 111] 
   Write to static field 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService.solver 
from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:from instance method new 
org.apache.hadoop.resourceestimator.service.ResourceEstimatorService() At 
ResourceEstimatorService.java:[line 92] 
   Found reliance on default encoding in 
org.apache.hadoop.resourceestimator.translator.impl.BaseLogParser.parseStream(InputStream):in
 
org.apache.hadoop.resourceestimator.translator.impl.BaseLogParser.parseStream(InputStream):
 new java.io.InputStreamReader(InputStream) At BaseLogParser.java:[line 104] 
   
org.apache.hadoop.resourceestimator.translator.impl.LogParserUtil.parseLog(String)
 may fail to clean up java.io.InputStream Obligation to clean up resource 
created at LogParserUtil.java:up java.io.InputStream Obligation to clean up 
resource created at LogParserUtil.java:[line 94] is not discharged 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager

[VOTE] Merge yarn-native-services branch into trunk

2017-10-30 Thread Jian He

Hi All,

I would like to restart the vote for merging yarn-native-services to trunk.
Since last vote, we have been working on several issues in documentation, DNS, 
CLI modifications etc. We believe now the feature is in a much better shape.

Some back ground:
At a high level, the following are the key feautres implemented.
- YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate 
existing services to YARN either docker or non-docker based.
- YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to deploy 
a service via a simple JSON spec
- YARN-4757[3]. Extending today's service registry with a simple DNS service to 
enable users to discover services deployed on YARN via standard DNS lookup
- YARN-6419[4]. UI support for native-services on the new YARN UI
All these new services are optional and are sitting outside of the existing 
system, and have no impact on existing system if disabled.

Special thanks to a team of folks who worked hard towards this: Billie Rinaldi, 
Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K S, Sunil G, 
Akhil PB, Eric Yang. This effort could not be possible without their ideas and 
hard work.
Also thanks Allen for some review and verifications.

Thanks,
Jian

[1] https://issues.apache.org/jira/browse/YARN-5079
[2] https://issues.apache.org/jira/browse/YARN-4793
[3] https://issues.apache.org/jira/browse/YARN-4757
[4] https://issues.apache.org/jira/browse/YARN-6419

Cutting branch-2.9

2017-10-30 Thread Subru Krishnan

We want to give heads up that we are going to cut branch-2.9 tomorrow
morning.

We are down to 3 blockers and they all are close to being committed (thanks
everyone):
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release

There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which are
close to completion.

Folks who are working/reviewing these, kindly prioritize accordingly so
that we can make the release on time.
https://issues.apache.org/jira/browse/YARN-7398?filter=12342468

Thanks in advance!

-Subru/Arun

[jira] [Resolved] (YARN-5326) Support for recurring reservations in the YARN ReservationSystem

2017-10-30 Thread Subru Krishnan (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Subru Krishnan resolved YARN-5326.
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
   3.0.0
   2.9.0

Marking this as resolved as all sub-tasks are closed.

> Support for recurring reservations in the YARN ReservationSystem
> 
>
> Key: YARN-5326
> URL: https://issues.apache.org/jira/browse/YARN-5326
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Carlo Curino
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: SupportRecurringReservationsInRayon.pdf
>
>
> YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle 
> time explicitly, i.e. users can now "reserve" capacity ahead of time which is 
> predictably allocated to them. Most SLA jobs/workflows are recurring so they 
> need the same resources periodically. With the current implementation, users 
> will have to make individual reservations for each run. This is an umbrella 
> JIRA to enhance the reservation system by adding native support for recurring 
> reservations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

2017-10-30 Thread Jitendra Pandey

Hi Konstantin,
 Thank you for taking out time to review ozone. I appreciate your comments and 
questions.

> There are two main limitations in HDFS
> a) The throughput of Namespace operations. Which is limited by the 
>number of RPCs the NameNode can handle
> b) The number of objects (files + blocks) the system can maintain. 
>Which is limited by the memory size of the NameNode.

   I agree completely. We believe ozone attempts to address both these issues 
for HDFS.

   Let us look at the Number of objects problem. Ozone directly addresses the 
scalability of number of blocks by introducing storage containers that can hold 
multiple blocks together. The earlier efforts on this were complicated by the 
fact that block manager and namespace are intertwined in HDFS Namenode. There 
have been efforts in past to separate block manager from namespace for e.g. 
HDFS-5477. Ozone addresses this problem by cleanly separating the block layer.  
Separation of block layer also addresses the file/directories scalability 
because it frees up the blockmap from the namenode.

   Separate block layer relieves namenode from handling block reports, IBRs, 
heartbeats, replication monitor etc, and thus reduces the contention on 
FSNamesystem lock and significantly reduces the GC pressure on the namenode. 
These improvements will greatly help the RPC performance of the Namenode.

> Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and 
> HDFS-8. The design doc for the new architecture has never been 
> published.

  We do believe that Namenode can leverage the ozone’s storage container layer, 
however, that is also a big effort. We would like to first have block layer 
stabilized in ozone before taking that up. However, we would certainly support 
any community effort on that, and in fact it was brought up in last BoF session 
at the summit.

  Big data is evolving rapidly. We see our customers needing scalable file 
systems, Objects stores(like S3) and Block Store(for docker and VMs). Ozone 
improves HDFS in two ways. It addresses throughput and scale issues of HDFS, 
and enriches it with newer capabilities.

> Ozone is a big enough system to deserve its own project.

I took a quick look at the core code in ozone and the cloc command reports 
22,511 lines of functionality changes in Java.

This patch also brings in web framework code like Angular.js and that brings in 
bunch of css and js files that contribute to the size of the patch, and the 
rest are test and documentation changes.

I hope this addresses your concerns.

Best regards,
jitendra

On 10/28/17, 2:00 PM, "Konstantin Shvachko"  wrote:

Hey guys,

It is an interesting question whether Ozone should be a part of Hadoop.
There are two main reasons why I think it should not.

1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
sizable community behind, it looks to me like a whole new project.
It is essentially a new storage system, with different (than HDFS)
architecture, separate S3-like APIs. This is really great - the World sure
needs more distributed file systems. But it is not clear why Ozone should
co-exist with HDFS under the same roof.

2. Ozone is probably just the first step in rebuilding HDFS under a new
architecture. With the next steps presumably being HDFS-10419 and
HDFS-8.
The design doc for the new architecture has never been published. I can
only assume based on some presentations and personal communications that
the idea is to use Ozone as a block storage, and re-implement NameNode, so
that it stores only a partial namesapce in memory, while the bulk of it
(cold data) is persisted to a local storage.
Such architecture makes me wonder if it solves Hadoop's main problems.
There are two main limitations in HDFS:
  a. The throughput of Namespace operations. Which is limited by the number
of RPCs the NameNode can handle
  b. The number of objects (files + blocks) the system can maintain. Which
is limited by the memory size of the NameNode.
The RPC performance (a) is more important for Hadoop scalability than the
object count (b). The read RPCs being the main priority.
The new architecture targets the object count problem, but in the expense
of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
Also based on the use patterns on our large clusters we read up to 90% of
the data we write, so cold data is a small fraction and most of it must be
cached.

To summarize:
- Ozone is a big enough system to deserve its own project.
- The architecture that Ozone leads to does not seem to solve the intrinsic
problems of current HDFS.

I will post my opinion in the Ozone jira. Should be more convenient to
discuss it there for further

[jira] [Resolved] (YARN-3813) Support Application timeout feature in YARN.

2017-10-30 Thread Rohith Sharma K S (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-3813.
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
   2.9.0

Closing this issue as all sub-items are done.

> Support Application timeout feature in YARN. 
> -
>
> Key: YARN-3813
> URL: https://issues.apache.org/jira/browse/YARN-3813
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: scheduler
>Reporter: nijel
>Assignee: Rohith Sharma K S
> Fix For: 2.9.0, 3.0.0
>
> Attachments: 0001-YARN-3813.patch, 0002_YARN-3813.patch, YARN 
> Application Timeout .pdf, Yarn Application Timeout.v1.pdf
>
>
> It will be useful to support Application Timeout in YARN. Some use cases are 
> not worried about the output of the applications if the application is not 
> completed in a specific time. 
> *Background:*
> The requirement is to show the CDR statistics of last few  minutes, say for 
> every 5 minutes. The same Job will run continuously with different dataset.
> So one job will be started in every 5 minutes. The estimate time for this 
> task is 2 minutes or lesser time. 
> If the application is not completing in the given time the output is not 
> useful.
> *Proposal*
> So idea is to support application timeout, with which timeout parameter is 
> given while submitting the job. 
> Here, user is expecting to finish (complete or kill) the application in the 
> given time.
> One option for us is to move this logic to Application client (who submit the 
> job). 
> But it will be nice if it can be generic logic and can make more robust.
> Kindly provide your suggestions/opinion on this feature. If it sounds good, i 
> will update the design doc and prototype patch



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org