[jira] [Commented] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766124#comment-13766124 ] Eli Collins commented on YARN-1024: --- bq. keeping virtual cores to express parallelism sounds good as it is clear it is not a real core. Hm, I read this the other way. If a framework asks for three vcores on a host it intends to run some code on three real physical cores at the same time. If a long lived framework wants to reserve 2 cores per host it would ask for 2 cores (and 100% YCU per core). Sandy's proposal, switching to cores and YCU instead of just vcores, is equivalent to the proposal above of getting rid of vcores and supporting fractional cores. A vcore becomes a core and YCU is just a way to express that you want a fraction of a core. Sounds good to me. Define a virtual core unambigiously --- Key: YARN-1024 URL: https://issues.apache.org/jira/browse/YARN-1024 Project: Hadoop YARN Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: CPUasaYARNresource.pdf We need to clearly define the meaning of a virtual core unambiguously so that it's easy to migrate applications between clusters. For e.g. here is Amazon EC2 definition of ECU: http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it Essentially we need to clearly define a YARN Virtual Core (YVC). Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-257) NM should gracefully handle a full local disk
[ https://issues.apache.org/jira/browse/YARN-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752764#comment-13752764 ] Eli Collins commented on YARN-257: -- This seems like a blocker for GA given that MR1 handles disk failures. NM should gracefully handle a full local disk - Key: YARN-257 URL: https://issues.apache.org/jira/browse/YARN-257 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.2-alpha, 0.23.5 Reporter: Jason Lowe When a local disk becomes full, the node will fail every container launched on it because the container is unable to localize. It tries to create an app-specific directory for each local and log directories. If any of those directory creates fail (due to lack of free space) the container fails. It would be nice if the node could continue to launch containers using the space available on other disks rather than failing all containers trying to launch on the node. This is somewhat related to YARN-91 but is centered around the disk becoming full rather than the disk failing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-91) DFIP aka 'NodeManager should handle Disk-Failures In Place'
[ https://issues.apache.org/jira/browse/YARN-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752763#comment-13752763 ] Eli Collins commented on YARN-91: - This seems like a blocker for GA given that MR1 handles disk failures. DFIP aka 'NodeManager should handle Disk-Failures In Place' - Key: YARN-91 URL: https://issues.apache.org/jira/browse/YARN-91 Project: Hadoop YARN Issue Type: Task Components: nodemanager Reporter: Vinod Kumar Vavilapalli Moving stuff over from the MAPREDUCE JIRA: MAPREDUCE-3121 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730961#comment-13730961 ] Eli Collins commented on YARN-1024: --- bq. vcores are optional anyway (only used in DRF) Sandy corrected me offline that while this is true for the CS it is not true for the FS, which by default (w/o DRF) will not schedule more containers worth of vcores than configured vcores (which seems like it could lead to under-utilization given that the default resource calculator only uses memory and not every container needs a whole core). By default the # vcores is the # cores on the machine and MR asks containers w/ 1 vcore so we effectively have vcore=pcore today as the default (re-inforced by the decision to remove the notion of pcore in YARN-782). Define a virtual core unambigiously --- Key: YARN-1024 URL: https://issues.apache.org/jira/browse/YARN-1024 Project: Hadoop YARN Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy We need to clearly define the meaning of a virtual core unambiguously so that it's easy to migrate applications between clusters. For e.g. here is Amazon EC2 definition of ECU: http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it Essentially we need to clearly define a YARN Virtual Core (YVC). Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.* -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730301#comment-13730301 ] Eli Collins commented on YARN-1024: --- I agree we need to define the meaning of a virtual core unambiguously (otherwise we won't be able to support two different frameworks on the same cluster that may have differing ideas of what a vcore is). I also agree with Phil that there are essentially two high-level use cases: 1. Jobs that want to express how much CPU capacity the job needs. Real world example - a distcp job wants to express it needs 100 containers but only a fraction of a CPU for each since it will spend most of its time blocking on IO. 2. Services - ie long-lived frameworks (ie support 2-level scheduling) - that want to request cores on many machines on a cluster and want to express CPU-level parallelism and aggregate demand (because they will schedule fine-grain requests w/in their long-lived containers). Eg a framework should be able to ask for two containers on a host, each with one core, so it can get two containers that can execute in parallel on a full core. This is assuming we plan to support long-running services in Yarn (YARN-896), which is hopefully not controversial. Real world example is HBase which may want 2 guaranteed cores per host on a given set of hosts. Seems like there are two high-level approaches: 1. Get rid of vcores. If we define 1vcore=1pcore (1vcore=1vcpu for virtual environments) and support fractional cores (YARN-972) then services can ask for 1 or more vcores knowing they're getting real cores and jobs just ask for what fraction of a vcore they think they need. This is really abandoning the concept of a virtual core because it's actually expressing a physical requirement (like memory, we assume Yarn is not dramatically over-committing the host). We can handle heterogeneous CPUs via attributes (as discussed in other Yarn jiras) since most clusters in my experience don't have wildly different processors (eg 1 or 2 generations is common), and attributes are sufficient to express policies like all my cores should have equal/comparable performance. 2. Keep going with vcores as a CPU unit of measurement. If we define 1vcore=1ECU (works 1:1 for virtual environments) then services (#1) need to understand the the power of a core so they can ask for that many vcores - essentially they are just undoing the virtualization. YARN would need to make sure two containers each with 1 pcores worth of vcores does in fact give you two cores( just like hypervisors schedule vcpus for the same VM on different pcores to ensure parallelism), but there would be no guarantee that two containers on the same host each w/ one vcore would run in parallel. Jobs that want fractional cores would just express 1vcore per container and work they're way up based on the experience running on the cluster (or also undo the virtualization by calculating vcore/pcore if they know what fraction of a pcore they want). Heterogenous CPUs does not fall out naturally (still need attributes) since there's no guarantee you can describe the difference between two CPUs is roughly 1 or more vcore (eg 2.4 - vs 2.0 Ghz 1ECU), however there's no need for fractional vcores. I think either is reasonable and can be made to work, though I think #1 is preferable because: - Some frameworks want to express containers in physical resources (this is consistent with how YARN handles memory) - You can support jobs that don't want a full core via fractional cores (or slightly over-committing cores) - You can support heterogeneous cores via attributes (I want equivalent containers) - vcores are optional anyway (only used in DRF) and therefore only need to be expressed if you care about physical cores because you need to reserve them or say you want a fraction of one Either way I think vcore is the wrong name either way because in #1 1vcore=1pcore so there's no virtualization and in #2 1 vcore is not a virtualization of a core (10 vcores does not give me 10 levels of parallelism), it's _just a unit_ (like an ECU). Define a virtual core unambigiously --- Key: YARN-1024 URL: https://issues.apache.org/jira/browse/YARN-1024 Project: Hadoop YARN Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy We need to clearly define the meaning of a virtual core unambiguously so that it's easy to migrate applications between clusters. For e.g. here is Amazon EC2 definition of ECU: http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it Essentially we need to clearly define a YARN Virtual Core (YVC). Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2
[jira] [Comment Edited] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730301#comment-13730301 ] Eli Collins edited comment on YARN-1024 at 8/6/13 3:09 AM: --- I agree we need to define the meaning of a virtual core unambiguously (otherwise we won't be able to support two different frameworks on the same cluster that may have differing ideas of what a vcore is). I also agree with Phil that there are essentially two high-level use cases: 1. Jobs that want to express how much CPU capacity the job needs. Real world example - a distcp job wants to express it needs 100 containers but only a fraction of a CPU for each since it will spend most of its time blocking on IO. 2. Services - ie long-lived frameworks (ie support 2-level scheduling) - that want to request cores on many machines on a cluster and want to express CPU-level parallelism and aggregate demand (because they will schedule fine-grain requests w/in their long-lived containers). Eg a framework should be able to ask for two containers on a host, each with one core, so it can get two containers that can execute in parallel on a full core. This is assuming we plan to support long-running services in Yarn (YARN-896), which is hopefully not controversial. Real world example is HBase which may want 2 guaranteed cores per host on a given set of hosts. Seems like there are two high-level approaches: 1. Get rid of vcores. If we define 1vcore=1pcore (1vcore=1vcpu for virtual environments) and support fractional cores (YARN-972) then services can ask for 1 or more vcores knowing they're getting real cores and jobs just ask for what fraction of a vcore they think they need. This is really abandoning the concept of a virtual core because it's actually expressing a physical requirement (like memory, we assume Yarn is not dramatically over-committing the host). We can handle heterogeneous CPUs via attributes (as discussed in other Yarn jiras) since most clusters in my experience don't have wildly different processors (eg 1 or 2 generations is common), and attributes are sufficient to express policies like all my cores should have equal/comparable performance. 2. Keep going with vcores as a CPU unit of measurement. If we define 1vcore=1ECU (works 1:1 for virtual environments) then services need to understand the the power of a core so they can ask for that many vcores - essentially they are just undoing the virtualization. YARN would need to make sure two containers each with 1 pcores worth of vcores does in fact give you two cores (just like hypervisors schedule vcpus for the same VM on different pcores to ensure parallelism), but there would be no guarantee that two containers on the same host each w/ one vcore would run in parallel. Jobs that want fractional cores would just express 1vcore per container and work their way up based on the experience running on the cluster (or also undo the virtualization by calculating vcore/pcore if they know what fraction of a pcore they want). Heterogenous CPUs does not fall out naturally (still need attributes) since there's no guarantee you can describe the difference between two CPUs is roughly 1 or more vcore (eg 2.4 - vs 2.0 Ghz 1ECU), however there's no need for fractional vcores. I think either is reasonable and can be made to work, though I think #1 is preferable because: - Some frameworks want to express containers in physical resources (this is consistent with how YARN handles memory) - You can support jobs that don't want a full core via fractional cores (or slightly over-committing cores) - You can support heterogeneous cores via attributes (I want equivalent containers) - vcores are optional anyway (only used in DRF) and therefore only need to be expressed if you care about physical cores because you need to reserve them or say you want a fraction of one Either way I think vcore is the wrong name because in #1 1vcore=1pcore so there's no virtualization and in #2 1 vcore is not a virtualization of a core (10 vcores does not give me 10 levels of parallelism), it's _just a unit_ (like an ECU). was (Author: eli): I agree we need to define the meaning of a virtual core unambiguously (otherwise we won't be able to support two different frameworks on the same cluster that may have differing ideas of what a vcore is). I also agree with Phil that there are essentially two high-level use cases: 1. Jobs that want to express how much CPU capacity the job needs. Real world example - a distcp job wants to express it needs 100 containers but only a fraction of a CPU for each since it will spend most of its time blocking on IO. 2. Services - ie long-lived frameworks (ie support 2-level scheduling) - that want to request cores on many machines on a cluster and want to express CPU-level parallelism and aggregate
[jira] [Comment Edited] (YARN-1024) Define a virtual core unambigiously
[ https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730301#comment-13730301 ] Eli Collins edited comment on YARN-1024 at 8/6/13 3:08 AM: --- I agree we need to define the meaning of a virtual core unambiguously (otherwise we won't be able to support two different frameworks on the same cluster that may have differing ideas of what a vcore is). I also agree with Phil that there are essentially two high-level use cases: 1. Jobs that want to express how much CPU capacity the job needs. Real world example - a distcp job wants to express it needs 100 containers but only a fraction of a CPU for each since it will spend most of its time blocking on IO. 2. Services - ie long-lived frameworks (ie support 2-level scheduling) - that want to request cores on many machines on a cluster and want to express CPU-level parallelism and aggregate demand (because they will schedule fine-grain requests w/in their long-lived containers). Eg a framework should be able to ask for two containers on a host, each with one core, so it can get two containers that can execute in parallel on a full core. This is assuming we plan to support long-running services in Yarn (YARN-896), which is hopefully not controversial. Real world example is HBase which may want 2 guaranteed cores per host on a given set of hosts. Seems like there are two high-level approaches: 1. Get rid of vcores. If we define 1vcore=1pcore (1vcore=1vcpu for virtual environments) and support fractional cores (YARN-972) then services can ask for 1 or more vcores knowing they're getting real cores and jobs just ask for what fraction of a vcore they think they need. This is really abandoning the concept of a virtual core because it's actually expressing a physical requirement (like memory, we assume Yarn is not dramatically over-committing the host). We can handle heterogeneous CPUs via attributes (as discussed in other Yarn jiras) since most clusters in my experience don't have wildly different processors (eg 1 or 2 generations is common), and attributes are sufficient to express policies like all my cores should have equal/comparable performance. 2. Keep going with vcores as a CPU unit of measurement. If we define 1vcore=1ECU (works 1:1 for virtual environments) then services need to understand the the power of a core so they can ask for that many vcores - essentially they are just undoing the virtualization. YARN would need to make sure two containers each with 1 pcores worth of vcores does in fact give you two cores (just like hypervisors schedule vcpus for the same VM on different pcores to ensure parallelism), but there would be no guarantee that two containers on the same host each w/ one vcore would run in parallel. Jobs that want fractional cores would just express 1vcore per container and work their way up based on the experience running on the cluster (or also undo the virtualization by calculating vcore/pcore if they know what fraction of a pcore they want). Heterogenous CPUs does not fall out naturally (still need attributes) since there's no guarantee you can describe the difference between two CPUs is roughly 1 or more vcore (eg 2.4 - vs 2.0 Ghz 1ECU), however there's no need for fractional vcores. I think either is reasonable and can be made to work, though I think #1 is preferable because: - Some frameworks want to express containers in physical resources (this is consistent with how YARN handles memory) - You can support jobs that don't want a full core via fractional cores (or slightly over-committing cores) - You can support heterogeneous cores via attributes (I want equivalent containers) - vcores are optional anyway (only used in DRF) and therefore only need to be expressed if you care about physical cores because you need to reserve them or say you want a fraction of one Either way I think vcore is the wrong name either way because in #1 1vcore=1pcore so there's no virtualization and in #2 1 vcore is not a virtualization of a core (10 vcores does not give me 10 levels of parallelism), it's _just a unit_ (like an ECU). was (Author: eli): I agree we need to define the meaning of a virtual core unambiguously (otherwise we won't be able to support two different frameworks on the same cluster that may have differing ideas of what a vcore is). I also agree with Phil that there are essentially two high-level use cases: 1. Jobs that want to express how much CPU capacity the job needs. Real world example - a distcp job wants to express it needs 100 containers but only a fraction of a CPU for each since it will spend most of its time blocking on IO. 2. Services - ie long-lived frameworks (ie support 2-level scheduling) - that want to request cores on many machines on a cluster and want to express CPU-level parallelism and
[jira] [Updated] (YARN-899) Get queue administration ACLs working
[ https://issues.apache.org/jira/browse/YARN-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated YARN-899: - Target Version/s: 2.1.0-beta Worth considering as a GA blocker since this is a regression vs MR1. Get queue administration ACLs working - Key: YARN-899 URL: https://issues.apache.org/jira/browse/YARN-899 Project: Hadoop YARN Issue Type: Bug Components: scheduler Affects Versions: 2.1.0-beta Reporter: Sandy Ryza The Capacity Scheduler documents the yarn.scheduler.capacity.root.queue-path.acl_administer_queue config option for controlling who can administer a queue, but it is not hooked up to anything. The Fair Scheduler could make use of a similar option as well. This is a feature-parity regression from MR1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-516) TestContainerLocalizer.testContainerLocalizerMain is failing
[ https://issues.apache.org/jira/browse/YARN-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621255#comment-13621255 ] Eli Collins commented on YARN-516: -- I reverted this change (and the initial HADOOP-9357 patch). We'll put this fix back in the HADOOP-9357 patch if we do another rev. TestContainerLocalizer.testContainerLocalizerMain is failing Key: YARN-516 URL: https://issues.apache.org/jira/browse/YARN-516 Project: Hadoop YARN Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: Andrew Wang Fix For: 2.0.5-beta Attachments: YARN-516.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (YARN-149) ZK-based High Availability (HA) for ResourceManager (RM)
[
https://issues.apache.org/jira/browse/YARN-149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins reopened YARN-149:
--
Assignee: (was: Bikas Saha)
ZK-based High Availability (HA) for ResourceManager (RM)
Key: YARN-149
URL: https://issues.apache.org/jira/browse/YARN-149
Project: Hadoop YARN
Issue Type: New Feature
Reporter: Harsh J
One of the goals presented on MAPREDUCE-279 was to have high availability.
One way that was discussed, per Mahadev/others on
https://issues.apache.org/jira/browse/MAPREDUCE-2648 and other places, was ZK:
{quote}
Am not sure, if you already know about the MR-279 branch (the next version of
MR framework). We've been trying to integrate ZK into the framework from the
beginning. As for now, we are just doing restart with ZK but soon we should
have a HA soln with ZK.
{quote}
There is now MAPREDUCE-4343 that tracks recoverability via ZK. This JIRA is
meant to track HA via ZK.
Currently there isn't a HA solution for RM, via ZK or otherwise.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-381) Improve FS docs
Eli Collins created YARN-381: Summary: Improve FS docs Key: YARN-381 URL: https://issues.apache.org/jira/browse/YARN-381 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Eli Collins Priority: Minor The MR2 FS docs could use some improvements. Configuration: - sizebasedweight - what is the size here? Total memory usage? Pool properties: - minResources - what does min amount of aggregate memory mean given that this is not a reservation? - maxResources - is this a hard limit? - weight: How is this ratio configured? Eg base is 1 and all weights are relative to that? - schedulingMode - what is the default? Is fifo pure fifo, eg waits until all tasks for the job are finished before launching the next job? There's no mention of ACLs, even though they're supported. See the CS docs for comparison. Also there are a couple typos worth fixing while we're at it, eg finish. apps to run Worth keeping in mind that some of these will need to be updated to reflect that resource calculators are now pluggable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-375) FIFO scheduler may crash due to bugg app
Eli Collins created YARN-375:
Summary: FIFO scheduler may crash due to bugg app
Key: YARN-375
URL: https://issues.apache.org/jira/browse/YARN-375
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Priority: Critical
The following code should check for a 0 return value rather than crash!
{code}
int availableContainers =
node.getAvailableResource().getMemory() / capability.getMemory(); //
TODO: A buggy
//
application
// with
this
// zero
would
//
crash the
//
scheduler.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-5) Add support for FifoScheduler to schedule CPU along with memory.
[ https://issues.apache.org/jira/browse/YARN-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated YARN-5: --- Issue Type: New Feature (was: Sub-task) Parent: (was: YARN-2) Add support for FifoScheduler to schedule CPU along with memory. Key: YARN-5 URL: https://issues.apache.org/jira/browse/YARN-5 Project: Hadoop YARN Issue Type: New Feature Reporter: Arun C Murthy Assignee: Arun C Murthy -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-340) Rename DefaultResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13568365#comment-13568365 ] Eli Collins commented on YARN-340: -- I like MemoryResourceCalculator (or MemoryBasedResourceCalculator) better as well, that was my first choice, and is what the attached patch uses. Arun went with that on YARN-2 but people objected, if people are OK with it now then let's go with it. I don't think proliferation of many class names will be an issue. Rename DefaultResourceCalculator Key: YARN-340 URL: https://issues.apache.org/jira/browse/YARN-340 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Attachments: yarn-340.txt, yarn-340.txt Let's rename DefaultResourceCalculator to something like MemoryResourceCalculator or SingleResourceCalculator. The default resource calculator is the one specified by yarn.scheduler.capacity.resource-calculator in yarn-default.xml (which may change). We can do this compatibly now since YARN-2 hasn't been released yet, but changing this later will be a pain if we ever make a different resource calculator the default (or DefaultResourceCalculator won't actually be the default, which is weird). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-2) Enhance CS to schedule accounting for both memory and cpu cores
[ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554149#comment-13554149 ] Eli Collins commented on YARN-2: bq. Could we change the name of ResourceMemoryCpuComparator to something more like DefaultMultiResourceComparator? I think ResourceMemoryCpuNetworkBandwithDiskStorageGPUComparator is a bit long, but it is the direction we are headed in. The problem with naming a class DefaultResource is that changing the default in the future is a pain. While I prefer MemoryResourceCalculator (it's explicit, and I don't think we'll see lots of different resource calculators) something like SingleResourceComparator fixes the naming issue and also won't have the issue of a new class for every policy. I filed YARN-340 to address this. Enhance CS to schedule accounting for both memory and cpu cores --- Key: YARN-2 URL: https://issues.apache.org/jira/browse/YARN-2 Project: Hadoop YARN Issue Type: New Feature Components: capacityscheduler, scheduler Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, YARN-2-help.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch With YARN being a general purpose system, it would be useful for several applications (MPI et al) to specify not just memory but also CPU (cores) for their resource requirements. Thus, it would be useful to the CapacityScheduler to account for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-340) Rename DefaultResourceCalculator
Eli Collins created YARN-340: Summary: Rename DefaultResourceCalculator Key: YARN-340 URL: https://issues.apache.org/jira/browse/YARN-340 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Eli Collins Priority: Minor Let's rename DefaultResourceCalculator to something like MemoryResourceCalculator or SingleResourceCalculator. The default resource calculator is the one specified by yarn.scheduler.capacity.resource-calculator in yarn-default.xml (which may change). We can do this compatibly now since YARN-2 hasn't been released yet, but changing this later will be a pain if we ever make a different resource calculator the default (or DefaultResourceCalculator won't actually be the default, which is weird). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-340) Rename DefaultResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated YARN-340: - Attachment: yarn-340.txt Patch attached. (Used svn mv for DefaultResourceCalculator to maintain history, so this patch shouldn't be checked in verbatim) Rename DefaultResourceCalculator Key: YARN-340 URL: https://issues.apache.org/jira/browse/YARN-340 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Eli Collins Priority: Minor Attachments: yarn-340.txt Let's rename DefaultResourceCalculator to something like MemoryResourceCalculator or SingleResourceCalculator. The default resource calculator is the one specified by yarn.scheduler.capacity.resource-calculator in yarn-default.xml (which may change). We can do this compatibly now since YARN-2 hasn't been released yet, but changing this later will be a pain if we ever make a different resource calculator the default (or DefaultResourceCalculator won't actually be the default, which is weird). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-340) Rename DefaultResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned YARN-340: Assignee: Eli Collins Rename DefaultResourceCalculator Key: YARN-340 URL: https://issues.apache.org/jira/browse/YARN-340 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Attachments: yarn-340.txt Let's rename DefaultResourceCalculator to something like MemoryResourceCalculator or SingleResourceCalculator. The default resource calculator is the one specified by yarn.scheduler.capacity.resource-calculator in yarn-default.xml (which may change). We can do this compatibly now since YARN-2 hasn't been released yet, but changing this later will be a pain if we ever make a different resource calculator the default (or DefaultResourceCalculator won't actually be the default, which is weird). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-340) Rename DefaultResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated YARN-340: - Attachment: yarn-340.txt Looks like svn diff after doing a mv and edit didn't generate a diff that test-patch could apply, this one should work. Rename DefaultResourceCalculator Key: YARN-340 URL: https://issues.apache.org/jira/browse/YARN-340 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Attachments: yarn-340.txt, yarn-340.txt Let's rename DefaultResourceCalculator to something like MemoryResourceCalculator or SingleResourceCalculator. The default resource calculator is the one specified by yarn.scheduler.capacity.resource-calculator in yarn-default.xml (which may change). We can do this compatibly now since YARN-2 hasn't been released yet, but changing this later will be a pain if we ever make a different resource calculator the default (or DefaultResourceCalculator won't actually be the default, which is weird). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-340) Rename DefaultResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553272#comment-13553272 ] Eli Collins commented on YARN-340: -- Release audit warnings are unrelated (YARN-341) Rename DefaultResourceCalculator Key: YARN-340 URL: https://issues.apache.org/jira/browse/YARN-340 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Attachments: yarn-340.txt, yarn-340.txt Let's rename DefaultResourceCalculator to something like MemoryResourceCalculator or SingleResourceCalculator. The default resource calculator is the one specified by yarn.scheduler.capacity.resource-calculator in yarn-default.xml (which may change). We can do this compatibly now since YARN-2 hasn't been released yet, but changing this later will be a pain if we ever make a different resource calculator the default (or DefaultResourceCalculator won't actually be the default, which is weird). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-340) Rename DefaultResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553289#comment-13553289 ] Eli Collins commented on YARN-340: -- Yea, that's why I suggested SingleResourceCalculator as well, what do you think of using it? I actually think you're original name was bedtter (and don't think we'll end up with that many permutations in practice which is I think MemoryResourceCalculator is best) but if people are concerned with that at least SingleResourceCalculator means we won't proliferate as many but we'll still be able to change the default in the future without renaming classes or having the DefaultResourceCalculator not actually be the default. Rename DefaultResourceCalculator Key: YARN-340 URL: https://issues.apache.org/jira/browse/YARN-340 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Attachments: yarn-340.txt, yarn-340.txt Let's rename DefaultResourceCalculator to something like MemoryResourceCalculator or SingleResourceCalculator. The default resource calculator is the one specified by yarn.scheduler.capacity.resource-calculator in yarn-default.xml (which may change). We can do this compatibly now since YARN-2 hasn't been released yet, but changing this later will be a pain if we ever make a different resource calculator the default (or DefaultResourceCalculator won't actually be the default, which is weird). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (YARN-340) Rename DefaultResourceCalculator
[ https://issues.apache.org/jira/browse/YARN-340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553289#comment-13553289 ] Eli Collins edited comment on YARN-340 at 1/14/13 11:55 PM: Yea, that's why I suggested SingleResourceCalculator as well, what do you think of using it? I actually think your original name was better (and don't think we'll end up with that many permutations in practice which is why I went with MemoryResourceCalculator) but if people are concerned with that at least SingleResourceCalculator means we won't proliferate as many but we'll still be able to change the default in the future without renaming classes or having the DefaultResourceCalculator not actually be the default. was (Author: eli): Yea, that's why I suggested SingleResourceCalculator as well, what do you think of using it? I actually think you're original name was bedtter (and don't think we'll end up with that many permutations in practice which is I think MemoryResourceCalculator is best) but if people are concerned with that at least SingleResourceCalculator means we won't proliferate as many but we'll still be able to change the default in the future without renaming classes or having the DefaultResourceCalculator not actually be the default. Rename DefaultResourceCalculator Key: YARN-340 URL: https://issues.apache.org/jira/browse/YARN-340 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Attachments: yarn-340.txt, yarn-340.txt Let's rename DefaultResourceCalculator to something like MemoryResourceCalculator or SingleResourceCalculator. The default resource calculator is the one specified by yarn.scheduler.capacity.resource-calculator in yarn-default.xml (which may change). We can do this compatibly now since YARN-2 hasn't been released yet, but changing this later will be a pain if we ever make a different resource calculator the default (or DefaultResourceCalculator won't actually be the default, which is weird). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-154) Create Yarn trunk and commit jobs
Eli Collins created YARN-154: Summary: Create Yarn trunk and commit jobs Key: YARN-154 URL: https://issues.apache.org/jira/browse/YARN-154 Project: Hadoop YARN Issue Type: Task Reporter: Eli Collins Yarn should have Hadoop-Yarn-trunk and Hadoop-Yarn-trunk-Commit jenkins jobs that correspond to the Common, HDFS, and MR ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-36) branch-2.1.0-alpha doesn't build
Eli Collins created YARN-36:
---
Summary: branch-2.1.0-alpha doesn't build
Key: YARN-36
URL: https://issues.apache.org/jira/browse/YARN-36
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.1.0-alpha
Reporter: Eli Collins
Priority: Blocker
branch-2.1.0-alpha doesn't build due to the following. Per YARN-1 I updated the
mvn version to be 2.1.0-SNAPSHOT, before I hit this issue it didn't compile due
to the bogus version.
{noformat}
hadoop-branch-2.1.0-alpha $ mvn compile
[INFO] Scanning for projects...
[ERROR] The build could not read 1 project - [Help 1]
[ERROR]
[ERROR] The project org.apache.hadoop:hadoop-yarn-project:2.1.0-SNAPSHOT
(/home/eli/src/hadoop-branch-2.1.0-alpha/hadoop-yarn-project/pom.xml) has 1
error
[ERROR] 'dependencies.dependency.version' for org.hsqldb:hsqldb:jar is
missing. @ line 160, column 17
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
