[jira] [Resolved] (HADOOP-3444) Implementing a Resource Manager (V1) for Hadoop

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J resolved HADOOP-3444.
-

Resolution: Duplicate

(Re-resolving as dupe)

> Implementing a Resource Manager (V1) for Hadoop
> ---
>
> Key: HADOOP-3444
> URL: https://issues.apache.org/jira/browse/HADOOP-3444
> Project: Hadoop Common
>  Issue Type: New Feature
>Reporter: Vivek Ratan
> Attachments: RMArch-V1.jpg
>
>
> HADOOP-3421 lists the requirements for a Resource Manager for Hadoop. This 
> Jira tracks its implementation. 
> It is expected that this Jira will be used to keep track of various other 
> Jiras that will be opened towards implementing Version 1 of the Resource 
> Manager. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-3444) Implementing a Resource Manager (V1) for Hadoop

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J resolved HADOOP-3444.
-

Resolution: Fixed

MAPREDUCE-279 has covered this. Resolving as dupe, same as its parent.

> Implementing a Resource Manager (V1) for Hadoop
> ---
>
> Key: HADOOP-3444
> URL: https://issues.apache.org/jira/browse/HADOOP-3444
> Project: Hadoop Common
>  Issue Type: New Feature
>Reporter: Vivek Ratan
> Attachments: RMArch-V1.jpg
>
>
> HADOOP-3421 lists the requirements for a Resource Manager for Hadoop. This 
> Jira tracks its implementation. 
> It is expected that this Jira will be used to keep track of various other 
> Jiras that will be opened towards implementing Version 1 of the Resource 
> Manager. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HADOOP-3444) Implementing a Resource Manager (V1) for Hadoop

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J reopened HADOOP-3444:
-


> Implementing a Resource Manager (V1) for Hadoop
> ---
>
> Key: HADOOP-3444
> URL: https://issues.apache.org/jira/browse/HADOOP-3444
> Project: Hadoop Common
>  Issue Type: New Feature
>Reporter: Vivek Ratan
> Attachments: RMArch-V1.jpg
>
>
> HADOOP-3421 lists the requirements for a Resource Manager for Hadoop. This 
> Jira tracks its implementation. 
> It is expected that this Jira will be used to keep track of various other 
> Jiras that will be opened towards implementing Version 1 of the Resource 
> Manager. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-3421) Requirements for a Resource Manager for Hadoop

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J resolved HADOOP-3421.
-

Resolution: Duplicate

Resolving as dupe of MAPREDUCE-279. Although, this is much better doc-wise and 
serves as a good reference.

Please reopen if I missed something that the other didn't provide (and was the 
goal here).

> Requirements for a Resource Manager for Hadoop
> --
>
> Key: HADOOP-3421
> URL: https://issues.apache.org/jira/browse/HADOOP-3421
> Project: Hadoop Common
>  Issue Type: New Feature
>Reporter: Vivek Ratan
>
> This is a proposal to extend the scheduling functionality of Hadoop to allow 
> sharing of large clusters without the use of HOD.  We're suffering from 
> performance issues with HOD and not finding it the right model for running 
> jobs. We have concluded that a native Hadoop Resource Manager would be more 
> useful to many people if it supported the features we need for sharing 
> clusters across large groups and organizations.
> Below are the key requirements for a Resource Manager for Hadoop. First, some 
> terminology used in this writeup: 
> * *RM*: Resource Manager. What we're building.
> * *MR*: Map Reduce.
> * A *job* is an MR job for now, but can be any request. Jobs are submitted by 
> users to the Grid. MR jobs are made up of units of computation called *tasks*.
> * A grid has a variety of *resources* of different *capacities* that are 
> allocated to tasks. For the the early version of the grid, the only resource 
> considered is a Map or Reduce slot, which can execute a task. Each slot can 
> run one or more tasks. Later versions may look at resources such as local 
> temporary storage or CPUs. 
> * *V1*: version 1. Some features are simplified for V1. 
> h3. Orgs, queues, users, jobs
> Organizations (*Orgs*) are distinct entities for administration, 
> configuration, billing and reporting purposes. *Users* belong to Orgs. Orgs 
> have *queues* of jobs, where a queue represents a collection of jobs that 
> share some scheduling criteria. 
>* *1.1.* For V1, each queue will belong to one Org and each Org will have 
> one queue. 
>* *1.2.* Jobs are submitted to queues. A single job can be submitted to 
> only one queue. It follows that a job will have a user and an Org associated 
> with it. 
>* *1.3.* A user can belong to multiple Orgs and can potentially submit 
> jobs to multiple queues. 
>* *1.4.* Orgs are guaranteed a fraction of the capacity of the grid (their 
> 'guaranteed capacity') in the sense that a certain capacity of resources will 
> be at their disposal. All jobs submitted to the queues of an Org will have 
> access to the capacity guaranteed to the Org. 
>   ** Note: it is expected that the sum of the guaranteed capacity of each 
> Org should equal the resources in the Grid. If the sum is lower, some 
> resources will not be used. If the sum is higher, the RM cannot maintain 
> guarantees for all Orgs. 
>* *1.5.* At any given time, free resources can be allocated to any Org 
> beyond their guaranteed capacity. For example this may be in the proportion 
> of guaranteed capacities of various Orgs or some other way. However, these 
> excess allocated resources can be reclaimed and made available to another Org 
>  in order to meet its capacity guarantee.
>* *1.6.* N minutes after an org reclaims resources, it should have all its 
> reserved capacity available. Put another way, the system will guarantee that 
> excess resources taken from an Org will be restored to it within N minutes of 
> its need for them.
>* *1.7.* Queues have access control. Queues can specify which users are 
> (not) allowed to submit jobs to it. A user's job submission will be rejected 
> if the user does not have access rights to the queue. 
> h3. Job capacity
>* *2.1.* Users will just submit jobs to the Grid. They do not need to 
> specify the capacity required for their jobs (i.e. how many parallel tasks 
> the job needs). [Most MR jobs are elastic and do not require a fixed number 
> of parallel tasks to run - they can run with as little or as much task 
> parallelism as they can get. This amount of task parallelism is usually 
> limited by the number of mappers required (which is computed by the system 
> and not by the user) or the amount of free resources available in the grid. 
> In most cases, the user wants to just submit a job and let the system take 
> care of utilizing as many or as little resources as it can.]
> h3. Priorities
>* *3.1.* Jobs can optionally have priorities associated with them. For V1, 
> we support the same set of priorities available to MR jobs today. 
>* *3.2.* Queues can optionally support priorities for jobs. By default, a 
> queue does not support priorities, in which case it will ignore (wi

Build failed in Jenkins: Hadoop-Common-0.23-Build #285

[Apache Jenkins Server]

See 

--
[...truncated 13277 lines...]
Running org.apache.hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs
Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.153 sec
Running org.apache.hadoop.fs.viewfs.TestViewFsLocalFs
Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.025 sec
Running org.apache.hadoop.fs.TestGlobPattern
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.149 sec
Running org.apache.hadoop.fs.TestS3_LocalFileContextURI
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.081 sec
Running org.apache.hadoop.fs.TestLocalFSFileContextCreateMkdir
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.705 sec
Running org.apache.hadoop.fs.TestHarFileSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.353 sec
Running org.apache.hadoop.fs.TestFileSystemCaching
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.734 sec
Running org.apache.hadoop.fs.TestLocalFsFCStatistics
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.553 sec
Running org.apache.hadoop.fs.TestHardLink
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.317 sec
Running org.apache.hadoop.fs.TestCommandFormat
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.165 sec
Running org.apache.hadoop.fs.TestLocal_S3FileContextURI
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.196 sec
Running org.apache.hadoop.fs.TestLocalFileSystem
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.84 sec
Running org.apache.hadoop.fs.TestFcLocalFsPermission
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.699 sec
Running org.apache.hadoop.fs.TestListFiles
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.593 sec
Running org.apache.hadoop.fs.TestPath
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.878 sec
Running org.apache.hadoop.fs.kfs.TestKosmosFileSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.606 sec
Running org.apache.hadoop.fs.TestGlobExpander
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.114 sec
Running org.apache.hadoop.fs.TestFilterFileSystem
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.645 sec
Running org.apache.hadoop.fs.TestFcLocalFsUtil
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.588 sec
Running org.apache.hadoop.fs.TestGetFileBlockLocations
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.759 sec
Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
Tests run: 29, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.832 sec
Running org.apache.hadoop.fs.s3.TestINode
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 sec
Running org.apache.hadoop.fs.s3.TestS3Credentials
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.217 sec
Running org.apache.hadoop.fs.s3.TestS3FileSystem
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.229 sec
Running org.apache.hadoop.fs.TestDU
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.266 sec
Running org.apache.hadoop.record.TestBuffer
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.115 sec
Running org.apache.hadoop.record.TestRecordVersioning
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.192 sec
Running org.apache.hadoop.record.TestRecordIO
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.216 sec
Running org.apache.hadoop.metrics2.source.TestJvmMetrics
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 sec
Running org.apache.hadoop.metrics2.util.TestSampleStat
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.133 sec
Running org.apache.hadoop.metrics2.util.TestMetricsCache
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.054 sec
Running org.apache.hadoop.metrics2.lib.TestInterns
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.276 sec
Running org.apache.hadoop.metrics2.lib.TestMetricsAnnotations
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.495 sec
Running org.apache.hadoop.metrics2.lib.TestMutableMetrics
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.453 sec
Running org.apache.hadoop.metrics2.lib.TestUniqNames
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.145 sec
Running org.apache.hadoop.metrics2.lib.TestMetricsRegistry
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.426 sec
Running org.apache.hadoop.metrics2.impl.TestMetricsCollectorImpl
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.288 sec
Running org.apache.hadoop.metrics2.impl.TestGangliaMetrics
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.472 sec
Running org.apache.hadoop.metrics2.impl.TestSinkQueue
Tests ru