[jira] [Created] (YARN-159) RM web ui applications page should be sorted to display last app first
Thomas Graves created YARN-159: -- Summary: RM web ui applications page should be sorted to display last app first Key: YARN-159 URL: https://issues.apache.org/jira/browse/YARN-159 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.4 Reporter: Thomas Graves RM web ui applications page should be sorted to display last app first. It currently sorts with smallest application id first, which is the first apps that were submitted. After you have one page worth of apps its much more useful for it to sort such that the biggest appid (last submitted app) shows up first. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-159) RM web ui applications page should be sorted to display last app first
[ https://issues.apache.org/jira/browse/YARN-159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476195#comment-13476195 ] Robert Joseph Evans commented on YARN-159: -- +1 for the idea. This bugs me all the time. > RM web ui applications page should be sorted to display last app first > --- > > Key: YARN-159 > URL: https://issues.apache.org/jira/browse/YARN-159 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.4 >Reporter: Thomas Graves > > RM web ui applications page should be sorted to display last app first. > It currently sorts with smallest application id first, which is the first > apps that were submitted. After you have one page worth of apps its much > more useful for it to sort such that the biggest appid (last submitted app) > shows up first. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-159) RM web ui applications page should be sorted to display last app first
[ https://issues.apache.org/jira/browse/YARN-159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned YARN-159: -- Assignee: Thomas Graves > RM web ui applications page should be sorted to display last app first > --- > > Key: YARN-159 > URL: https://issues.apache.org/jira/browse/YARN-159 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.4 >Reporter: Thomas Graves >Assignee: Thomas Graves > > RM web ui applications page should be sorted to display last app first. > It currently sorts with smallest application id first, which is the first > apps that were submitted. After you have one page worth of apps its much > more useful for it to sort such that the biggest appid (last submitted app) > shows up first. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-159) RM web ui applications page should be sorted to display last app first
[ https://issues.apache.org/jira/browse/YARN-159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-159: --- Attachment: YARN-159.patch > RM web ui applications page should be sorted to display last app first > --- > > Key: YARN-159 > URL: https://issues.apache.org/jira/browse/YARN-159 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.4 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-159.patch > > > RM web ui applications page should be sorted to display last app first. > It currently sorts with smallest application id first, which is the first > apps that were submitted. After you have one page worth of apps its much > more useful for it to sort such that the biggest appid (last submitted app) > shows up first. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-159) RM web ui applications page should be sorted to display last app first
[ https://issues.apache.org/jira/browse/YARN-159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476205#comment-13476205 ] Thomas Graves commented on YARN-159: Sorry type - no should say is "now" descending. > RM web ui applications page should be sorted to display last app first > --- > > Key: YARN-159 > URL: https://issues.apache.org/jira/browse/YARN-159 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.4 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-159.patch > > > RM web ui applications page should be sorted to display last app first. > It currently sorts with smallest application id first, which is the first > apps that were submitted. After you have one page worth of apps its much > more useful for it to sort such that the biggest appid (last submitted app) > shows up first. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-159) RM web ui applications page should be sorted to display last app first
[ https://issues.apache.org/jira/browse/YARN-159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476218#comment-13476218 ] Hadoop QA commented on YARN-159: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549153/YARN-159.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/88//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/88//console This message is automatically generated. > RM web ui applications page should be sorted to display last app first > --- > > Key: YARN-159 > URL: https://issues.apache.org/jira/browse/YARN-159 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.4 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-159.patch > > > RM web ui applications page should be sorted to display last app first. > It currently sorts with smallest application id first, which is the first > apps that were submitted. After you have one page worth of apps its much > more useful for it to sort such that the biggest appid (last submitted app) > shows up first. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-147) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/YARN-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476220#comment-13476220 ] Hadoop QA commented on YARN-147: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549094/YARN-147-v1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/89//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/89//console This message is automatically generated. > Add support for CPU isolation/monitoring of containers > -- > > Key: YARN-147 > URL: https://issues.apache.org/jira/browse/YARN-147 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Andrew Ferguson > Fix For: 2.0.3-alpha > > Attachments: YARN-147-v1.patch, YARN-3.patch > > > This is a clone for YARN-3 to be able to submit the patch as YARN-3 does not > show the SUBMIT PATCH button. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-160) nodemanagers should obtain cpu/memory values from underlying OS
Alejandro Abdelnur created YARN-160: --- Summary: nodemanagers should obtain cpu/memory values from underlying OS Key: YARN-160 URL: https://issues.apache.org/jira/browse/YARN-160 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.0.3-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.3-alpha As mentioned in YARN-2 *NM memory and CPU configs* Currently these values are coming from the config of the NM, we should be able to obtain those values from the OS (ie, in the case of Linux from /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have an interface that obtains this information. In addition implementations of this interface should be able to specify a mem/cpu offset (amount of mem/cpu not to be avail as YARN resource), this would allow to reserve mem/cpu for the OS and other services outside of YARN containers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-2) Enhance CS to schedule accounting for both memory and cpu cores
[ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476251#comment-13476251 ] Alejandro Abdelnur commented on YARN-2: --- *Num cores...* I'd prefer to use a float instead of a int because it will avoid user confusion on what the integer part of the number means, specially when dealing with clusters with large number of cores in their nodes. *NM memory and CPU configs* Currently these values are coming from the config of the NM, we should be able to obtain those values from the OS (ie, in the case of Linux from /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have an interface that obtains this information. In addition implementations of this interface should be able to specify a mem/cpu offset (amount of mem/cpu not to be avail as YARN resource), this would allow to reserve mem/cpu for the OS and other services outside of YARN containers. I did a quick search and I couldn't find a JIRA for this, opening YARN-160. > Enhance CS to schedule accounting for both memory and cpu cores > --- > > Key: YARN-2 > URL: https://issues.apache.org/jira/browse/YARN-2 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacityscheduler, scheduler >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 2.0.3-alpha > > Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, > MAPREDUCE-4327.patch, MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, > MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, YARN-2-help.patch, > YARN-2.patch, YARN-2.patch > > > With YARN being a general purpose system, it would be useful for several > applications (MPI et al) to specify not just memory but also CPU (cores) for > their resource requirements. Thus, it would be useful to the > CapacityScheduler to account for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-147) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/YARN-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476268#comment-13476268 ] Alejandro Abdelnur commented on YARN-147: - Andrew, thanks for updating the patch. Mostly minor stuff: * CgroupsLCEResourcesHandler has an unused import for FileNotFoundException * CgroupsLCEResourcesHandler, on the double '//', on the init() method, trim the '/' is present at the beginning or end of the cgroupPrefix * CgroupsLCEResourcesHandler, parseMtab, no need have a local var for the fReader, the FileReader creation could be done directly in the BufferedReader constructor. * CgroupsLCEResourcesHandler, parseMtab, the last exception should print MTAB_FILE instead the BufferReader instance. * CgroupsLCEResourcesHandler, on parseMtab error I think we should thrown an exception to halt things. My reasoning is that if I've configured things to use cgroups, I'd expect them to be working as opposed to have a warning message lost in the logs. This would help identify misconfigurations. > Add support for CPU isolation/monitoring of containers > -- > > Key: YARN-147 > URL: https://issues.apache.org/jira/browse/YARN-147 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Andrew Ferguson > Fix For: 2.0.3-alpha > > Attachments: YARN-147-v1.patch, YARN-3.patch > > > This is a clone for YARN-3 to be able to submit the patch as YARN-3 does not > show the SUBMIT PATCH button. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-160) nodemanagers should obtain cpu/memory values from underlying OS
[ https://issues.apache.org/jira/browse/YARN-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476288#comment-13476288 ] Radim Kolar commented on YARN-160: -- there is a interface from obtaining these values from OS. Its called resourcecalculator plugin. It might be good to do autodetection but i need ability to override these values in config. > nodemanagers should obtain cpu/memory values from underlying OS > --- > > Key: YARN-160 > URL: https://issues.apache.org/jira/browse/YARN-160 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.3-alpha > > > As mentioned in YARN-2 > *NM memory and CPU configs* > Currently these values are coming from the config of the NM, we should be > able to obtain those values from the OS (ie, in the case of Linux from > /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have > an interface that obtains this information. In addition implementations of > this interface should be able to specify a mem/cpu offset (amount of mem/cpu > not to be avail as YARN resource), this would allow to reserve mem/cpu for > the OS and other services outside of YARN containers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-154) Create Yarn trunk and commit jobs
[ https://issues.apache.org/jira/browse/YARN-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476296#comment-13476296 ] Eli Collins commented on YARN-154: -- A single commit job works for me. > Create Yarn trunk and commit jobs > - > > Key: YARN-154 > URL: https://issues.apache.org/jira/browse/YARN-154 > Project: Hadoop YARN > Issue Type: Task >Reporter: Eli Collins > > Yarn should have Hadoop-Yarn-trunk and Hadoop-Yarn-trunk-Commit jenkins jobs > that correspond to the Common, HDFS, and MR ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (YARN-160) nodemanagers should obtain cpu/memory values from underlying OS
thanks Radim, unless i'm missing something this is not wired, correct? yes being able to override makes sense Alejandro On Oct 15, 2012, at 10:36 AM, "Radim Kolar (JIRA)" wrote: > >[ > https://issues.apache.org/jira/browse/YARN-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476288#comment-13476288 > ] > > Radim Kolar commented on YARN-160: > -- > > there is a interface from obtaining these values from OS. Its called > resourcecalculator plugin. It might be good to do autodetection but i need > ability to override these values in config. > >> nodemanagers should obtain cpu/memory values from underlying OS >> --- >> >>Key: YARN-160 >>URL: https://issues.apache.org/jira/browse/YARN-160 >>Project: Hadoop YARN >> Issue Type: Improvement >> Components: nodemanager >> Affects Versions: 2.0.3-alpha >> Reporter: Alejandro Abdelnur >> Assignee: Alejandro Abdelnur >>Fix For: 2.0.3-alpha >> >> >> As mentioned in YARN-2 >> *NM memory and CPU configs* >> Currently these values are coming from the config of the NM, we should be >> able to obtain those values from the OS (ie, in the case of Linux from >> /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should >> have an interface that obtains this information. In addition implementations >> of this interface should be able to specify a mem/cpu offset (amount of >> mem/cpu not to be avail as YARN resource), this would allow to reserve >> mem/cpu for the OS and other services outside of YARN containers. > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA administrators > For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-154) Create Yarn trunk and commit jobs
[ https://issues.apache.org/jira/browse/YARN-154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans reassigned YARN-154: Assignee: Robert Joseph Evans > Create Yarn trunk and commit jobs > - > > Key: YARN-154 > URL: https://issues.apache.org/jira/browse/YARN-154 > Project: Hadoop YARN > Issue Type: Task >Reporter: Eli Collins >Assignee: Robert Joseph Evans > > Yarn should have Hadoop-Yarn-trunk and Hadoop-Yarn-trunk-Commit jenkins jobs > that correspond to the Common, HDFS, and MR ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-160) nodemanagers should obtain cpu/memory values from underlying OS
[ https://issues.apache.org/jira/browse/YARN-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476330#comment-13476330 ] Alejandro Abdelnur commented on YARN-160: - Radim, thanks for pointing that. This it seems it is not wired for the NMs to report such info back to the RM, looking at the code NodeManager creates a NodeStatusUpdater using values from configuration files. Only the ContainersMonitorIMpl reads the underlying memory. So this JIRA would be to get the NodeStatusUpdater to use a resourcecalculator. Makes sense? > nodemanagers should obtain cpu/memory values from underlying OS > --- > > Key: YARN-160 > URL: https://issues.apache.org/jira/browse/YARN-160 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.3-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.3-alpha > > > As mentioned in YARN-2 > *NM memory and CPU configs* > Currently these values are coming from the config of the NM, we should be > able to obtain those values from the OS (ie, in the case of Linux from > /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have > an interface that obtains this information. In addition implementations of > this interface should be able to specify a mem/cpu offset (amount of mem/cpu > not to be avail as YARN resource), this would allow to reserve mem/cpu for > the OS and other services outside of YARN containers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-154) Create Yarn trunk and commit jobs
[ https://issues.apache.org/jira/browse/YARN-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476331#comment-13476331 ] Robert Joseph Evans commented on YARN-154: -- I grabbed this to at least get a first version out. Eli could you take a look at https://builds.apache.org/job/Hadoop-Yarn-trunk/ and tell me if everything looks OK. I copied most of it from Hadoop-Hdfs-trunk. I'll update the e-mail addresses to be yarn-dev, assuming that everything goes OK. > Create Yarn trunk and commit jobs > - > > Key: YARN-154 > URL: https://issues.apache.org/jira/browse/YARN-154 > Project: Hadoop YARN > Issue Type: Task >Reporter: Eli Collins >Assignee: Robert Joseph Evans > > Yarn should have Hadoop-Yarn-trunk and Hadoop-Yarn-trunk-Commit jenkins jobs > that correspond to the Common, HDFS, and MR ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-154) Create Yarn trunk and commit jobs
[ https://issues.apache.org/jira/browse/YARN-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476358#comment-13476358 ] Eli Collins commented on YARN-154: -- Bobby, lgtm > Create Yarn trunk and commit jobs > - > > Key: YARN-154 > URL: https://issues.apache.org/jira/browse/YARN-154 > Project: Hadoop YARN > Issue Type: Task >Reporter: Eli Collins >Assignee: Robert Joseph Evans > > Yarn should have Hadoop-Yarn-trunk and Hadoop-Yarn-trunk-Commit jenkins jobs > that correspond to the Common, HDFS, and MR ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-154) Create Yarn trunk and commit jobs
[ https://issues.apache.org/jira/browse/YARN-154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476462#comment-13476462 ] Robert Joseph Evans commented on YARN-154: -- I keep getting an error of {noformat} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-yarn-server-nodemanager: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ? -> [Help 1] {noformat} Have you seen anything like this before? > Create Yarn trunk and commit jobs > - > > Key: YARN-154 > URL: https://issues.apache.org/jira/browse/YARN-154 > Project: Hadoop YARN > Issue Type: Task >Reporter: Eli Collins >Assignee: Robert Joseph Evans > > Yarn should have Hadoop-Yarn-trunk and Hadoop-Yarn-trunk-Commit jenkins jobs > that correspond to the Common, HDFS, and MR ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira