[jira] [Commented] (YARN-2976) Invalid docs for specifying yarn.nodemanager.docker-container-executor.exec-name
[ https://issues.apache.org/jira/browse/YARN-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498487#comment-14498487 ] Vijay Bhat commented on YARN-2976: -- [~hitesh], yes that makes sense. I can call the new config property yarn.nodemanager.docker-container-executor.exec-cmd. I would think that this setting should take precedence over yarn.nodemanager.docker-container-executor.exec-name in case both are present, what do you think? > Invalid docs for specifying > yarn.nodemanager.docker-container-executor.exec-name > > > Key: YARN-2976 > URL: https://issues.apache.org/jira/browse/YARN-2976 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 2.6.0 >Reporter: Hitesh Shah >Assignee: Vijay Bhat >Priority: Minor > > Docs on > http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html > mention setting "docker -H=tcp://0.0.0.0:4243" for > yarn.nodemanager.docker-container-executor.exec-name. > However, the actual implementation does a fileExists for the specified value. > Either the docs need to be fixed or the impl changed to allow relative paths > or commands with additional args -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2976) Invalid docs for specifying yarn.nodemanager.docker-container-executor.exec-name
[ https://issues.apache.org/jira/browse/YARN-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498389#comment-14498389 ] Vijay Bhat commented on YARN-2976: -- [~hitesh], I can take this on. Would your preference be for fixing the documentation or extending the current behavior? I would think it makes more sense to have a flexible configuration. > Invalid docs for specifying > yarn.nodemanager.docker-container-executor.exec-name > > > Key: YARN-2976 > URL: https://issues.apache.org/jira/browse/YARN-2976 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 2.6.0 >Reporter: Hitesh Shah >Assignee: Vijay Bhat >Priority: Minor > > Docs on > http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html > mention setting "docker -H=tcp://0.0.0.0:4243" for > yarn.nodemanager.docker-container-executor.exec-name. > However, the actual implementation does a fileExists for the specified value. > Either the docs need to be fixed or the impl changed to allow relative paths > or commands with additional args -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-2976) Invalid docs for specifying yarn.nodemanager.docker-container-executor.exec-name
[ https://issues.apache.org/jira/browse/YARN-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat reassigned YARN-2976: Assignee: Vijay Bhat > Invalid docs for specifying > yarn.nodemanager.docker-container-executor.exec-name > > > Key: YARN-2976 > URL: https://issues.apache.org/jira/browse/YARN-2976 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 2.6.0 >Reporter: Hitesh Shah >Assignee: Vijay Bhat >Priority: Minor > > Docs on > http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html > mention setting "docker -H=tcp://0.0.0.0:4243" for > yarn.nodemanager.docker-container-executor.exec-name. > However, the actual implementation does a fileExists for the specified value. > Either the docs need to be fixed or the impl changed to allow relative paths > or commands with additional args -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371786#comment-14371786 ] Vijay Bhat commented on YARN-2828: -- [~aw], I've gotten all the tests to pass again for this patch. Could you please review when you get a chance? Thanks! > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch, YARN-2828.002.patch, > YARN-2828.003.patch, YARN-2828.004.patch, YARN-2828.005.patch, > YARN-2828.006.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: YARN-2828.006.patch > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch, YARN-2828.002.patch, > YARN-2828.003.patch, YARN-2828.004.patch, YARN-2828.005.patch, > YARN-2828.006.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: YARN-2828.005.patch > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch, YARN-2828.002.patch, > YARN-2828.003.patch, YARN-2828.004.patch, YARN-2828.005.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: YARN-2828.004.patch > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch, YARN-2828.002.patch, > YARN-2828.003.patch, YARN-2828.004.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: (was: HADOOP-9329.004.patch) > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch, YARN-2828.002.patch, > YARN-2828.003.patch, YARN-2828.004.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: HADOOP-9329.004.patch > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: HADOOP-9329.004.patch, YARN-2828.001.patch, > YARN-2828.002.patch, YARN-2828.003.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: YARN-2828.003.patch > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch, YARN-2828.002.patch, > YARN-2828.003.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: (was: YARN-2828.002.patch) > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: YARN-2828.002.patch > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch, YARN-2828.002.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: YARN-2828.002.patch > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch, YARN-2828.002.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: (was: YARN-2828.002.patch) > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343737#comment-14343737 ] Vijay Bhat commented on YARN-2828: -- [~aw], thanks for the review. I wasn't able to find any other instances of html parameter sanitization in the code base, which is why I added jsoup to the pom file. I've moved the version definition to the hadoop-project pom file as you suggested. Please let me know if there are any other mods you'd like to see. > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch, YARN-2828.002.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: YARN-2828.002.patch > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch, YARN-2828.002.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-545) NodeResourceMonitor and its Impl are emty and may be removed
[ https://issues.apache.org/jira/browse/YARN-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-545: Attachment: YARN-545.001.patch > NodeResourceMonitor and its Impl are emty and may be removed > > > Key: YARN-545 > URL: https://issues.apache.org/jira/browse/YARN-545 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-545.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-545) NodeResourceMonitor and its Impl are emty and may be removed
[ https://issues.apache.org/jira/browse/YARN-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat reassigned YARN-545: --- Assignee: Vijay Bhat > NodeResourceMonitor and its Impl are emty and may be removed > > > Key: YARN-545 > URL: https://issues.apache.org/jira/browse/YARN-545 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Vijay Bhat >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: YARN-2828.001.patch > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: (was: YARN-2828.001.patch) > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: YARN-2828.001.patch > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: (was: YARN-2828.001.patch) > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-2828: - Attachment: YARN-2828.001.patch > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-2828.001.patch > > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-2828) Enable auto refresh of web pages (using http parameter)
[ https://issues.apache.org/jira/browse/YARN-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat reassigned YARN-2828: Assignee: Vijay Bhat > Enable auto refresh of web pages (using http parameter) > --- > > Key: YARN-2828 > URL: https://issues.apache.org/jira/browse/YARN-2828 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Tim Robertson >Assignee: Vijay Bhat >Priority: Minor > > The MR1 Job Tracker had a useful HTTP parameter of e.g. "&refresh=3" that > could be appended to URLs which enabled a page reload. This was very useful > when developing mapreduce jobs, especially to watch counters changing. This > is lost in the the Yarn interface. > Could be implemented as a page element (e.g. drop down or so), but I'd > recommend that the page not be more cluttered, and simply bring back the > optional "refresh" HTTP param. It worked really nicely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2230) Fix description of yarn.scheduler.maximum-allocation-vcores in yarn-default.xml (or code)
[
https://issues.apache.org/jira/browse/YARN-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265677#comment-14265677
]
Vijay Bhat commented on YARN-2230:
--
Thanks Jian! I've submitted the updated patch.
Patch summary:
I've updated the text description for the following properties in
yarn-default.xml to be reflective of the actual behavior in the code.
* yarn.scheduler.minimum-allocation-vcores
* yarn.scheduler.maximum-allocation-vcores
* yarn.scheduler.minimum-allocation-mb
* yarn.scheduler.maximum-allocation-mb
This is a documentation patch and does not change any code behavior.
> Fix description of yarn.scheduler.maximum-allocation-vcores in
> yarn-default.xml (or code)
> -
>
> Key: YARN-2230
> URL: https://issues.apache.org/jira/browse/YARN-2230
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, documentation, scheduler
>Affects Versions: 2.4.0
>Reporter: Adam Kawa
>Assignee: Vijay Bhat
>Priority: Minor
> Attachments: YARN-2230.001.patch, YARN-2230.002.patch
>
>
> When a user requests more vcores than the allocation limit (e.g.
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores), then
> InvalidResourceRequestException is thrown -
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
> {code}
> if (resReq.getCapability().getVirtualCores() < 0 ||
> resReq.getCapability().getVirtualCores() >
> maximumResource.getVirtualCores()) {
> throw new InvalidResourceRequestException("Invalid resource request"
> + ", requested virtual cores < 0"
> + ", or requested virtual cores > max configured"
> + ", requestedVirtualCores="
> + resReq.getCapability().getVirtualCores()
> + ", maxVirtualCores=" + maximumResource.getVirtualCores());
> }
> {code}
> According to documentation - yarn-default.xml
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml,
> the request should be capped to the allocation limit.
> {code}
>
> The maximum allocation for every container request at the RM,
> in terms of virtual CPU cores. Requests higher than this won't take
> effect,
> and will get capped to this value.
> yarn.scheduler.maximum-allocation-vcores
> 32
>
> {code}
> This means that:
> * Either documentation or code should be corrected (unless this exception is
> handled elsewhere accordingly, but it looks that it is not).
> This behavior is confusing, because when such a job (with
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores) is submitted, it does not make any
> progress. The warnings/exceptions are thrown at the scheduler (RM) side e.g.
> {code}
> 2014-06-29 00:34:51,469 WARN
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Invalid resource ask by application appattempt_1403993411503_0002_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, requested virtual cores < 0, or requested virtual cores >
> max configured, requestedVirtualCores=32, maxVirtualCores=3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:237)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:80)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:420)
> .
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> {code}
> * IMHO, such an exception should be forwarded to client. Otherwise, it is non
> obvious to discover why a job does not make any progress.
> The same looks to be related to memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-2230) Fix description of yarn.scheduler.maximum-allocation-vcores in yarn-default.xml (or code)
[
https://issues.apache.org/jira/browse/YARN-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vijay Bhat reassigned YARN-2230:
Assignee: Vijay Bhat
> Fix description of yarn.scheduler.maximum-allocation-vcores in
> yarn-default.xml (or code)
> -
>
> Key: YARN-2230
> URL: https://issues.apache.org/jira/browse/YARN-2230
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, documentation, scheduler
>Affects Versions: 2.4.0
>Reporter: Adam Kawa
>Assignee: Vijay Bhat
>Priority: Minor
> Attachments: YARN-2230.001.patch, YARN-2230.002.patch
>
>
> When a user requests more vcores than the allocation limit (e.g.
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores), then
> InvalidResourceRequestException is thrown -
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
> {code}
> if (resReq.getCapability().getVirtualCores() < 0 ||
> resReq.getCapability().getVirtualCores() >
> maximumResource.getVirtualCores()) {
> throw new InvalidResourceRequestException("Invalid resource request"
> + ", requested virtual cores < 0"
> + ", or requested virtual cores > max configured"
> + ", requestedVirtualCores="
> + resReq.getCapability().getVirtualCores()
> + ", maxVirtualCores=" + maximumResource.getVirtualCores());
> }
> {code}
> According to documentation - yarn-default.xml
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml,
> the request should be capped to the allocation limit.
> {code}
>
> The maximum allocation for every container request at the RM,
> in terms of virtual CPU cores. Requests higher than this won't take
> effect,
> and will get capped to this value.
> yarn.scheduler.maximum-allocation-vcores
> 32
>
> {code}
> This means that:
> * Either documentation or code should be corrected (unless this exception is
> handled elsewhere accordingly, but it looks that it is not).
> This behavior is confusing, because when such a job (with
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores) is submitted, it does not make any
> progress. The warnings/exceptions are thrown at the scheduler (RM) side e.g.
> {code}
> 2014-06-29 00:34:51,469 WARN
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Invalid resource ask by application appattempt_1403993411503_0002_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, requested virtual cores < 0, or requested virtual cores >
> max configured, requestedVirtualCores=32, maxVirtualCores=3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:237)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:80)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:420)
> .
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> {code}
> * IMHO, such an exception should be forwarded to client. Otherwise, it is non
> obvious to discover why a job does not make any progress.
> The same looks to be related to memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2230) Fix description of yarn.scheduler.maximum-allocation-vcores in yarn-default.xml (or code)
[
https://issues.apache.org/jira/browse/YARN-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vijay Bhat updated YARN-2230:
-
Attachment: YARN-2230.002.patch
> Fix description of yarn.scheduler.maximum-allocation-vcores in
> yarn-default.xml (or code)
> -
>
> Key: YARN-2230
> URL: https://issues.apache.org/jira/browse/YARN-2230
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, documentation, scheduler
>Affects Versions: 2.4.0
>Reporter: Adam Kawa
>Priority: Minor
> Attachments: YARN-2230.001.patch, YARN-2230.002.patch
>
>
> When a user requests more vcores than the allocation limit (e.g.
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores), then
> InvalidResourceRequestException is thrown -
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
> {code}
> if (resReq.getCapability().getVirtualCores() < 0 ||
> resReq.getCapability().getVirtualCores() >
> maximumResource.getVirtualCores()) {
> throw new InvalidResourceRequestException("Invalid resource request"
> + ", requested virtual cores < 0"
> + ", or requested virtual cores > max configured"
> + ", requestedVirtualCores="
> + resReq.getCapability().getVirtualCores()
> + ", maxVirtualCores=" + maximumResource.getVirtualCores());
> }
> {code}
> According to documentation - yarn-default.xml
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml,
> the request should be capped to the allocation limit.
> {code}
>
> The maximum allocation for every container request at the RM,
> in terms of virtual CPU cores. Requests higher than this won't take
> effect,
> and will get capped to this value.
> yarn.scheduler.maximum-allocation-vcores
> 32
>
> {code}
> This means that:
> * Either documentation or code should be corrected (unless this exception is
> handled elsewhere accordingly, but it looks that it is not).
> This behavior is confusing, because when such a job (with
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores) is submitted, it does not make any
> progress. The warnings/exceptions are thrown at the scheduler (RM) side e.g.
> {code}
> 2014-06-29 00:34:51,469 WARN
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Invalid resource ask by application appattempt_1403993411503_0002_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, requested virtual cores < 0, or requested virtual cores >
> max configured, requestedVirtualCores=32, maxVirtualCores=3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:237)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:80)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:420)
> .
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> {code}
> * IMHO, such an exception should be forwarded to client. Otherwise, it is non
> obvious to discover why a job does not make any progress.
> The same looks to be related to memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2230) Fix description of yarn.scheduler.maximum-allocation-vcores in yarn-default.xml (or code)
[
https://issues.apache.org/jira/browse/YARN-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265276#comment-14265276
]
Vijay Bhat commented on YARN-2230:
--
Jian,
I (Vijay Bhat) can definitely take care of that. Also, since this is my first
submission, I wanted to clarify - is the protocol that I assign the JIRA to
myself once I submit the patch? Apologies for any confusion. Thanks!
> Fix description of yarn.scheduler.maximum-allocation-vcores in
> yarn-default.xml (or code)
> -
>
> Key: YARN-2230
> URL: https://issues.apache.org/jira/browse/YARN-2230
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, documentation, scheduler
>Affects Versions: 2.4.0
>Reporter: Adam Kawa
>Priority: Minor
> Attachments: YARN-2230.001.patch
>
>
> When a user requests more vcores than the allocation limit (e.g.
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores), then
> InvalidResourceRequestException is thrown -
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
> {code}
> if (resReq.getCapability().getVirtualCores() < 0 ||
> resReq.getCapability().getVirtualCores() >
> maximumResource.getVirtualCores()) {
> throw new InvalidResourceRequestException("Invalid resource request"
> + ", requested virtual cores < 0"
> + ", or requested virtual cores > max configured"
> + ", requestedVirtualCores="
> + resReq.getCapability().getVirtualCores()
> + ", maxVirtualCores=" + maximumResource.getVirtualCores());
> }
> {code}
> According to documentation - yarn-default.xml
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml,
> the request should be capped to the allocation limit.
> {code}
>
> The maximum allocation for every container request at the RM,
> in terms of virtual CPU cores. Requests higher than this won't take
> effect,
> and will get capped to this value.
> yarn.scheduler.maximum-allocation-vcores
> 32
>
> {code}
> This means that:
> * Either documentation or code should be corrected (unless this exception is
> handled elsewhere accordingly, but it looks that it is not).
> This behavior is confusing, because when such a job (with
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores) is submitted, it does not make any
> progress. The warnings/exceptions are thrown at the scheduler (RM) side e.g.
> {code}
> 2014-06-29 00:34:51,469 WARN
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Invalid resource ask by application appattempt_1403993411503_0002_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, requested virtual cores < 0, or requested virtual cores >
> max configured, requestedVirtualCores=32, maxVirtualCores=3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:237)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:80)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:420)
> .
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> {code}
> * IMHO, such an exception should be forwarded to client. Otherwise, it is non
> obvious to discover why a job does not make any progress.
> The same looks to be related to memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2230) Fix description of yarn.scheduler.maximum-allocation-vcores in yarn-default.xml (or code)
[
https://issues.apache.org/jira/browse/YARN-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vijay Bhat updated YARN-2230:
-
Attachment: (was: YARN-2230.001.patch)
> Fix description of yarn.scheduler.maximum-allocation-vcores in
> yarn-default.xml (or code)
> -
>
> Key: YARN-2230
> URL: https://issues.apache.org/jira/browse/YARN-2230
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, documentation, scheduler
>Affects Versions: 2.4.0
>Reporter: Adam Kawa
>Priority: Minor
> Attachments: YARN-2230.001.patch
>
>
> When a user requests more vcores than the allocation limit (e.g.
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores), then
> InvalidResourceRequestException is thrown -
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
> {code}
> if (resReq.getCapability().getVirtualCores() < 0 ||
> resReq.getCapability().getVirtualCores() >
> maximumResource.getVirtualCores()) {
> throw new InvalidResourceRequestException("Invalid resource request"
> + ", requested virtual cores < 0"
> + ", or requested virtual cores > max configured"
> + ", requestedVirtualCores="
> + resReq.getCapability().getVirtualCores()
> + ", maxVirtualCores=" + maximumResource.getVirtualCores());
> }
> {code}
> According to documentation - yarn-default.xml
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml,
> the request should be capped to the allocation limit.
> {code}
>
> The maximum allocation for every container request at the RM,
> in terms of virtual CPU cores. Requests higher than this won't take
> effect,
> and will get capped to this value.
> yarn.scheduler.maximum-allocation-vcores
> 32
>
> {code}
> This means that:
> * Either documentation or code should be corrected (unless this exception is
> handled elsewhere accordingly, but it looks that it is not).
> This behavior is confusing, because when such a job (with
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores) is submitted, it does not make any
> progress. The warnings/exceptions are thrown at the scheduler (RM) side e.g.
> {code}
> 2014-06-29 00:34:51,469 WARN
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Invalid resource ask by application appattempt_1403993411503_0002_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, requested virtual cores < 0, or requested virtual cores >
> max configured, requestedVirtualCores=32, maxVirtualCores=3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:237)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:80)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:420)
> .
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> {code}
> * IMHO, such an exception should be forwarded to client. Otherwise, it is non
> obvious to discover why a job does not make any progress.
> The same looks to be related to memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2230) Fix description of yarn.scheduler.maximum-allocation-vcores in yarn-default.xml (or code)
[
https://issues.apache.org/jira/browse/YARN-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vijay Bhat updated YARN-2230:
-
Attachment: YARN-2230.001.patch
> Fix description of yarn.scheduler.maximum-allocation-vcores in
> yarn-default.xml (or code)
> -
>
> Key: YARN-2230
> URL: https://issues.apache.org/jira/browse/YARN-2230
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, documentation, scheduler
>Affects Versions: 2.4.0
>Reporter: Adam Kawa
>Priority: Minor
> Attachments: YARN-2230.001.patch
>
>
> When a user requests more vcores than the allocation limit (e.g.
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores), then
> InvalidResourceRequestException is thrown -
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
> {code}
> if (resReq.getCapability().getVirtualCores() < 0 ||
> resReq.getCapability().getVirtualCores() >
> maximumResource.getVirtualCores()) {
> throw new InvalidResourceRequestException("Invalid resource request"
> + ", requested virtual cores < 0"
> + ", or requested virtual cores > max configured"
> + ", requestedVirtualCores="
> + resReq.getCapability().getVirtualCores()
> + ", maxVirtualCores=" + maximumResource.getVirtualCores());
> }
> {code}
> According to documentation - yarn-default.xml
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml,
> the request should be capped to the allocation limit.
> {code}
>
> The maximum allocation for every container request at the RM,
> in terms of virtual CPU cores. Requests higher than this won't take
> effect,
> and will get capped to this value.
> yarn.scheduler.maximum-allocation-vcores
> 32
>
> {code}
> This means that:
> * Either documentation or code should be corrected (unless this exception is
> handled elsewhere accordingly, but it looks that it is not).
> This behavior is confusing, because when such a job (with
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores) is submitted, it does not make any
> progress. The warnings/exceptions are thrown at the scheduler (RM) side e.g.
> {code}
> 2014-06-29 00:34:51,469 WARN
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Invalid resource ask by application appattempt_1403993411503_0002_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, requested virtual cores < 0, or requested virtual cores >
> max configured, requestedVirtualCores=32, maxVirtualCores=3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:237)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:80)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:420)
> .
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> {code}
> * IMHO, such an exception should be forwarded to client. Otherwise, it is non
> obvious to discover why a job does not make any progress.
> The same looks to be related to memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2230) Fix description of yarn.scheduler.maximum-allocation-vcores in yarn-default.xml (or code)
[
https://issues.apache.org/jira/browse/YARN-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vijay Bhat updated YARN-2230:
-
Component/s: documentation
> Fix description of yarn.scheduler.maximum-allocation-vcores in
> yarn-default.xml (or code)
> -
>
> Key: YARN-2230
> URL: https://issues.apache.org/jira/browse/YARN-2230
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, documentation, scheduler
>Affects Versions: 2.4.0
>Reporter: Adam Kawa
>Priority: Minor
> Attachments: YARN-2230.001.patch
>
>
> When a user requests more vcores than the allocation limit (e.g.
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores), then
> InvalidResourceRequestException is thrown -
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
> {code}
> if (resReq.getCapability().getVirtualCores() < 0 ||
> resReq.getCapability().getVirtualCores() >
> maximumResource.getVirtualCores()) {
> throw new InvalidResourceRequestException("Invalid resource request"
> + ", requested virtual cores < 0"
> + ", or requested virtual cores > max configured"
> + ", requestedVirtualCores="
> + resReq.getCapability().getVirtualCores()
> + ", maxVirtualCores=" + maximumResource.getVirtualCores());
> }
> {code}
> According to documentation - yarn-default.xml
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml,
> the request should be capped to the allocation limit.
> {code}
>
> The maximum allocation for every container request at the RM,
> in terms of virtual CPU cores. Requests higher than this won't take
> effect,
> and will get capped to this value.
> yarn.scheduler.maximum-allocation-vcores
> 32
>
> {code}
> This means that:
> * Either documentation or code should be corrected (unless this exception is
> handled elsewhere accordingly, but it looks that it is not).
> This behavior is confusing, because when such a job (with
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores) is submitted, it does not make any
> progress. The warnings/exceptions are thrown at the scheduler (RM) side e.g.
> {code}
> 2014-06-29 00:34:51,469 WARN
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Invalid resource ask by application appattempt_1403993411503_0002_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, requested virtual cores < 0, or requested virtual cores >
> max configured, requestedVirtualCores=32, maxVirtualCores=3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:237)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:80)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:420)
> .
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> {code}
> * IMHO, such an exception should be forwarded to client. Otherwise, it is non
> obvious to discover why a job does not make any progress.
> The same looks to be related to memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2230) Fix description of yarn.scheduler.maximum-allocation-vcores in yarn-default.xml (or code)
[
https://issues.apache.org/jira/browse/YARN-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vijay Bhat updated YARN-2230:
-
Attachment: YARN-2230.001.patch
> Fix description of yarn.scheduler.maximum-allocation-vcores in
> yarn-default.xml (or code)
> -
>
> Key: YARN-2230
> URL: https://issues.apache.org/jira/browse/YARN-2230
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client, scheduler
>Affects Versions: 2.4.0
>Reporter: Adam Kawa
>Priority: Minor
> Attachments: YARN-2230.001.patch
>
>
> When a user requests more vcores than the allocation limit (e.g.
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores), then
> InvalidResourceRequestException is thrown -
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
> {code}
> if (resReq.getCapability().getVirtualCores() < 0 ||
> resReq.getCapability().getVirtualCores() >
> maximumResource.getVirtualCores()) {
> throw new InvalidResourceRequestException("Invalid resource request"
> + ", requested virtual cores < 0"
> + ", or requested virtual cores > max configured"
> + ", requestedVirtualCores="
> + resReq.getCapability().getVirtualCores()
> + ", maxVirtualCores=" + maximumResource.getVirtualCores());
> }
> {code}
> According to documentation - yarn-default.xml
> http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml,
> the request should be capped to the allocation limit.
> {code}
>
> The maximum allocation for every container request at the RM,
> in terms of virtual CPU cores. Requests higher than this won't take
> effect,
> and will get capped to this value.
> yarn.scheduler.maximum-allocation-vcores
> 32
>
> {code}
> This means that:
> * Either documentation or code should be corrected (unless this exception is
> handled elsewhere accordingly, but it looks that it is not).
> This behavior is confusing, because when such a job (with
> mapreduce.map.cpu.vcores is larger than
> yarn.scheduler.maximum-allocation-vcores) is submitted, it does not make any
> progress. The warnings/exceptions are thrown at the scheduler (RM) side e.g.
> {code}
> 2014-06-29 00:34:51,469 WARN
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> Invalid resource ask by application appattempt_1403993411503_0002_01
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
> resource request, requested virtual cores < 0, or requested virtual cores >
> max configured, requestedVirtualCores=32, maxVirtualCores=3
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:237)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:80)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:420)
> .
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
> {code}
> * IMHO, such an exception should be forwarded to client. Otherwise, it is non
> obvious to discover why a job does not make any progress.
> The same looks to be related to memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
