Build failed in Jenkins: Hadoop-Common-0.23-Build #285
See https://builds.apache.org/job/Hadoop-Common-0.23-Build/285/ -- [...truncated 13277 lines...] Running org.apache.hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.153 sec Running org.apache.hadoop.fs.viewfs.TestViewFsLocalFs Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.025 sec Running org.apache.hadoop.fs.TestGlobPattern Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.149 sec Running org.apache.hadoop.fs.TestS3_LocalFileContextURI Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.081 sec Running org.apache.hadoop.fs.TestLocalFSFileContextCreateMkdir Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.705 sec Running org.apache.hadoop.fs.TestHarFileSystem Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.353 sec Running org.apache.hadoop.fs.TestFileSystemCaching Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.734 sec Running org.apache.hadoop.fs.TestLocalFsFCStatistics Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.553 sec Running org.apache.hadoop.fs.TestHardLink Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.317 sec Running org.apache.hadoop.fs.TestCommandFormat Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.165 sec Running org.apache.hadoop.fs.TestLocal_S3FileContextURI Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.196 sec Running org.apache.hadoop.fs.TestLocalFileSystem Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.84 sec Running org.apache.hadoop.fs.TestFcLocalFsPermission Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.699 sec Running org.apache.hadoop.fs.TestListFiles Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.593 sec Running org.apache.hadoop.fs.TestPath Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.878 sec Running org.apache.hadoop.fs.kfs.TestKosmosFileSystem Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.606 sec Running org.apache.hadoop.fs.TestGlobExpander Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.114 sec Running org.apache.hadoop.fs.TestFilterFileSystem Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.645 sec Running org.apache.hadoop.fs.TestFcLocalFsUtil Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.588 sec Running org.apache.hadoop.fs.TestGetFileBlockLocations Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.759 sec Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract Tests run: 29, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.832 sec Running org.apache.hadoop.fs.s3.TestINode Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 sec Running org.apache.hadoop.fs.s3.TestS3Credentials Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.217 sec Running org.apache.hadoop.fs.s3.TestS3FileSystem Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.229 sec Running org.apache.hadoop.fs.TestDU Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.266 sec Running org.apache.hadoop.record.TestBuffer Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.115 sec Running org.apache.hadoop.record.TestRecordVersioning Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.192 sec Running org.apache.hadoop.record.TestRecordIO Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.216 sec Running org.apache.hadoop.metrics2.source.TestJvmMetrics Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 sec Running org.apache.hadoop.metrics2.util.TestSampleStat Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.133 sec Running org.apache.hadoop.metrics2.util.TestMetricsCache Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.054 sec Running org.apache.hadoop.metrics2.lib.TestInterns Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.276 sec Running org.apache.hadoop.metrics2.lib.TestMetricsAnnotations Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.495 sec Running org.apache.hadoop.metrics2.lib.TestMutableMetrics Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.453 sec Running org.apache.hadoop.metrics2.lib.TestUniqNames Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.145 sec Running org.apache.hadoop.metrics2.lib.TestMetricsRegistry Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.426 sec Running org.apache.hadoop.metrics2.impl.TestMetricsCollectorImpl Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.288 sec Running org.apache.hadoop.metrics2.impl.TestGangliaMetrics Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.472 sec Running org.apache.hadoop.metrics2.impl.TestSinkQueue Tests
[jira] [Resolved] (HADOOP-3421) Requirements for a Resource Manager for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-3421. - Resolution: Duplicate Resolving as dupe of MAPREDUCE-279. Although, this is much better doc-wise and serves as a good reference. Please reopen if I missed something that the other didn't provide (and was the goal here). Requirements for a Resource Manager for Hadoop -- Key: HADOOP-3421 URL: https://issues.apache.org/jira/browse/HADOOP-3421 Project: Hadoop Common Issue Type: New Feature Reporter: Vivek Ratan This is a proposal to extend the scheduling functionality of Hadoop to allow sharing of large clusters without the use of HOD. We're suffering from performance issues with HOD and not finding it the right model for running jobs. We have concluded that a native Hadoop Resource Manager would be more useful to many people if it supported the features we need for sharing clusters across large groups and organizations. Below are the key requirements for a Resource Manager for Hadoop. First, some terminology used in this writeup: * *RM*: Resource Manager. What we're building. * *MR*: Map Reduce. * A *job* is an MR job for now, but can be any request. Jobs are submitted by users to the Grid. MR jobs are made up of units of computation called *tasks*. * A grid has a variety of *resources* of different *capacities* that are allocated to tasks. For the the early version of the grid, the only resource considered is a Map or Reduce slot, which can execute a task. Each slot can run one or more tasks. Later versions may look at resources such as local temporary storage or CPUs. * *V1*: version 1. Some features are simplified for V1. h3. Orgs, queues, users, jobs Organizations (*Orgs*) are distinct entities for administration, configuration, billing and reporting purposes. *Users* belong to Orgs. Orgs have *queues* of jobs, where a queue represents a collection of jobs that share some scheduling criteria. * *1.1.* For V1, each queue will belong to one Org and each Org will have one queue. * *1.2.* Jobs are submitted to queues. A single job can be submitted to only one queue. It follows that a job will have a user and an Org associated with it. * *1.3.* A user can belong to multiple Orgs and can potentially submit jobs to multiple queues. * *1.4.* Orgs are guaranteed a fraction of the capacity of the grid (their 'guaranteed capacity') in the sense that a certain capacity of resources will be at their disposal. All jobs submitted to the queues of an Org will have access to the capacity guaranteed to the Org. ** Note: it is expected that the sum of the guaranteed capacity of each Org should equal the resources in the Grid. If the sum is lower, some resources will not be used. If the sum is higher, the RM cannot maintain guarantees for all Orgs. * *1.5.* At any given time, free resources can be allocated to any Org beyond their guaranteed capacity. For example this may be in the proportion of guaranteed capacities of various Orgs or some other way. However, these excess allocated resources can be reclaimed and made available to another Org in order to meet its capacity guarantee. * *1.6.* N minutes after an org reclaims resources, it should have all its reserved capacity available. Put another way, the system will guarantee that excess resources taken from an Org will be restored to it within N minutes of its need for them. * *1.7.* Queues have access control. Queues can specify which users are (not) allowed to submit jobs to it. A user's job submission will be rejected if the user does not have access rights to the queue. h3. Job capacity * *2.1.* Users will just submit jobs to the Grid. They do not need to specify the capacity required for their jobs (i.e. how many parallel tasks the job needs). [Most MR jobs are elastic and do not require a fixed number of parallel tasks to run - they can run with as little or as much task parallelism as they can get. This amount of task parallelism is usually limited by the number of mappers required (which is computed by the system and not by the user) or the amount of free resources available in the grid. In most cases, the user wants to just submit a job and let the system take care of utilizing as many or as little resources as it can.] h3. Priorities * *3.1.* Jobs can optionally have priorities associated with them. For V1, we support the same set of priorities available to MR jobs today. * *3.2.* Queues can optionally support priorities for jobs. By default, a queue does not support priorities, in which case it will ignore (with a warning) any priority levels specified by jobs submitted to it. If
[jira] [Reopened] (HADOOP-3444) Implementing a Resource Manager (V1) for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J reopened HADOOP-3444: - Implementing a Resource Manager (V1) for Hadoop --- Key: HADOOP-3444 URL: https://issues.apache.org/jira/browse/HADOOP-3444 Project: Hadoop Common Issue Type: New Feature Reporter: Vivek Ratan Attachments: RMArch-V1.jpg HADOOP-3421 lists the requirements for a Resource Manager for Hadoop. This Jira tracks its implementation. It is expected that this Jira will be used to keep track of various other Jiras that will be opened towards implementing Version 1 of the Resource Manager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-3444) Implementing a Resource Manager (V1) for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-3444. - Resolution: Fixed MAPREDUCE-279 has covered this. Resolving as dupe, same as its parent. Implementing a Resource Manager (V1) for Hadoop --- Key: HADOOP-3444 URL: https://issues.apache.org/jira/browse/HADOOP-3444 Project: Hadoop Common Issue Type: New Feature Reporter: Vivek Ratan Attachments: RMArch-V1.jpg HADOOP-3421 lists the requirements for a Resource Manager for Hadoop. This Jira tracks its implementation. It is expected that this Jira will be used to keep track of various other Jiras that will be opened towards implementing Version 1 of the Resource Manager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-3444) Implementing a Resource Manager (V1) for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-3444. - Resolution: Duplicate (Re-resolving as dupe) Implementing a Resource Manager (V1) for Hadoop --- Key: HADOOP-3444 URL: https://issues.apache.org/jira/browse/HADOOP-3444 Project: Hadoop Common Issue Type: New Feature Reporter: Vivek Ratan Attachments: RMArch-V1.jpg HADOOP-3421 lists the requirements for a Resource Manager for Hadoop. This Jira tracks its implementation. It is expected that this Jira will be used to keep track of various other Jiras that will be opened towards implementing Version 1 of the Resource Manager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
