[jira] [Commented] (YARN-7594) TestNMWebServices.testGetNMResourceInfo fails on trunk
[
https://issues.apache.org/jira/browse/YARN-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274664#comment-16274664
]
Gergely Novák commented on YARN-7594:
-
Attached a trivial patch to solve this.
> TestNMWebServices.testGetNMResourceInfo fails on trunk
> --
>
> Key: YARN-7594
> URL: https://issues.apache.org/jira/browse/YARN-7594
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager, webapp
>Reporter: Gergely Novák
>Assignee: Gergely Novák
> Attachments: YARN-7594.001.patch
>
>
> {code}
> // Access resource-2 should fail (null NMResourceInfo returned).
> JSONObject json = response.getEntity(JSONObject.class);
> assertIncludesException(json);
> {code}
> fails with
> {noformat}
> java.lang.AssertionError ...
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices.assertIncludesException(TestNMWebServices.java:541)
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices.testGetNMResourceInfo(TestNMWebServices.java:461)...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6633) Backport YARN-4167 to branch 2.7
[ https://issues.apache.org/jira/browse/YARN-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-6633: -- Description: NPE on RMActiveServices#serviceStop when store is null > Backport YARN-4167 to branch 2.7 > > > Key: YARN-6633 > URL: https://issues.apache.org/jira/browse/YARN-6633 > Project: Hadoop YARN > Issue Type: Task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Attachments: YARN-4167-branch-2.7.patch > > > NPE on RMActiveServices#serviceStop when store is null -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6632) Backport YARN-3425 to branch 2.7
[ https://issues.apache.org/jira/browse/YARN-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-6632: -- Description: NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed > Backport YARN-3425 to branch 2.7 > > > Key: YARN-6632 > URL: https://issues.apache.org/jira/browse/YARN-6632 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Attachments: YARN-3425-branch-2.7.patch > > > NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit > failed -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7242) Support specify values of different resource types in DistributedShell for easier testing
[ https://issues.apache.org/jira/browse/YARN-7242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-7242: Attachment: YARN-7242.003.patch > Support specify values of different resource types in DistributedShell for > easier testing > - > > Key: YARN-7242 > URL: https://issues.apache.org/jira/browse/YARN-7242 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Gergely Novák >Priority: Critical > Labels: newbie > Attachments: YARN-7242.001.patch, YARN-7242.002.patch, > YARN-7242.003.patch > > > Currently, DS supports specify resource profile, it's better to allow user to > directly specify resource keys/values from command line. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7675) The new UI won't load for pre 2.8 Hadoop versions because queueCapacitiesByPartition is missing from the scheduler API
Gergely Novák created YARN-7675:
---
Summary: The new UI won't load for pre 2.8 Hadoop versions because
queueCapacitiesByPartition is missing from the scheduler API
Key: YARN-7675
URL: https://issues.apache.org/jira/browse/YARN-7675
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-ui-v2
Reporter: Gergely Novák
If we connect the new YARN UI to any Hadoop versions older than 2.8 it won't
load. The console shows this trace:
{noformat}
TypeError: Cannot read property 'queueCapacitiesByPartition' of undefined
at Class.normalizeSingleResponse (yarn-ui.js:13903)
at Class.superWrapper [as normalizeSingleResponse] (vendor.js:31811)
at Class.handleQueue (yarn-ui.js:13928)
at Class.normalizeArrayResponse (yarn-ui.js:13952)
at Class.normalizeQueryResponse (vendor.js:101566)
at Class.normalizeResponse (vendor.js:101468)
at ember$data$lib$system$store$serializer$response$$normalizeResponseHelper
(vendor.js:95345)
at vendor.js:95672
at Backburner.run (vendor.js:10426)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6632) Backport YARN-3425 to branch 2.7
[ https://issues.apache.org/jira/browse/YARN-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295997#comment-16295997 ] Íñigo Goiri commented on YARN-6632: --- The whitespaces seem to be coming from yarn-default. I think this is OK. > Backport YARN-3425 to branch 2.7 > > > Key: YARN-6632 > URL: https://issues.apache.org/jira/browse/YARN-6632 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Attachments: YARN-3425-branch-2.7.patch > > > NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit > failed -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5150) [YARN-3368] Add a sunburst chart view for queues/applications resource usage to new YARN UI
[ https://issues.apache.org/jira/browse/YARN-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-5150: Attachment: (was: Screen Shot 2017-03-31 at 15.06.49.png) > [YARN-3368] Add a sunburst chart view for queues/applications resource usage > to new YARN UI > --- > > Key: YARN-5150 > URL: https://issues.apache.org/jira/browse/YARN-5150 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Gergely Novák > Attachments: YARN-5150.001.patch, YARN-5150.002.patch > > > An example of sunburst chart: https://bl.ocks.org/kerryrodden/7090426. > If we can introduce it to YARN UI, admins can easily get understanding of > relative resource usages and configured capacities for queues/applications. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5150) [YARN-3368] Add a sunburst chart view for queues/applications resource usage to new YARN UI
[ https://issues.apache.org/jira/browse/YARN-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-5150: Attachment: YARN-5150.002.patch Rebased to the latest trunk, updated the colors and the tooltip opacity to match the new style and. > [YARN-3368] Add a sunburst chart view for queues/applications resource usage > to new YARN UI > --- > > Key: YARN-5150 > URL: https://issues.apache.org/jira/browse/YARN-5150 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Gergely Novák > Attachments: Screen Shot 2017-03-31 at 15.06.49.png, > YARN-5150.001.patch, YARN-5150.002.patch > > > An example of sunburst chart: https://bl.ocks.org/kerryrodden/7090426. > If we can introduce it to YARN UI, admins can easily get understanding of > relative resource usages and configured capacities for queues/applications. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5150) [YARN-3368] Add a sunburst chart view for queues/applications resource usage to new YARN UI
[ https://issues.apache.org/jira/browse/YARN-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-5150: Attachment: Screen Shot 2017-11-17 at 12.05.28.png > [YARN-3368] Add a sunburst chart view for queues/applications resource usage > to new YARN UI > --- > > Key: YARN-5150 > URL: https://issues.apache.org/jira/browse/YARN-5150 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Gergely Novák > Attachments: Screen Shot 2017-11-17 at 12.05.28.png, > YARN-5150.001.patch, YARN-5150.002.patch > > > An example of sunburst chart: https://bl.ocks.org/kerryrodden/7090426. > If we can introduce it to YARN UI, admins can easily get understanding of > relative resource usages and configured capacities for queues/applications. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6483) Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat
[ https://issues.apache.org/jira/browse/YARN-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Rodríguez Hortalá updated YARN-6483: - Attachment: YARN-6483.002.patch > Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes > returned by the Resource Manager as a response to the Application Master > heartbeat > > > Key: YARN-6483 > URL: https://issues.apache.org/jira/browse/YARN-6483 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Juan Rodríguez Hortalá > Attachments: YARN-6483-v1.patch, YARN-6483.002.patch > > > The DECOMMISSIONING node state is currently used as part of the graceful > decommissioning mechanism to give time for tasks to complete in a node that > is scheduled for decommission, and for reducer tasks to read the shuffle > blocks in that node. Also, YARN effectively blacklists nodes in > DECOMMISSIONING state by assigning them a capacity of 0, to prevent > additional containers to be launched in those nodes, so no more shuffle > blocks are written to the node. This blacklisting is not effective for > applications like Spark, because a Spark executor running in a YARN container > will keep receiving more tasks after the corresponding node has been > blacklisted at the YARN level. We would like to propose a modification of the > YARN heartbeat mechanism so nodes transitioning to DECOMMISSIONING are added > to the list of updated nodes returned by the Resource Manager as a response > to the Application Master heartbeat. This way a Spark application master > would be able to blacklist a DECOMMISSIONING at the Spark level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7475) container log link is not working in new YARN UI
[
https://issues.apache.org/jira/browse/YARN-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248616#comment-16248616
]
Íñigo Goiri commented on YARN-7475:
---
Using the {{nodeHttpAddr}} looks like the way to go and it fixes the issue.
Do we still need the {{host}} field? In that case, should it be updated with
the {{nodeHttpAddr}} host part?
> container log link is not working in new YARN UI
>
>
> Key: YARN-7475
> URL: https://issues.apache.org/jira/browse/YARN-7475
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-7475.001.patch
>
>
> Container log link is broken
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7475) container log link is not working in new YARN UI
[ https://issues.apache.org/jira/browse/YARN-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248915#comment-16248915 ] Íñigo Goiri commented on YARN-7475: --- That sounds good. The clean up could be done later. Tested it in a small cluster and it looks good. +1 > container log link is not working in new YARN UI > > > Key: YARN-7475 > URL: https://issues.apache.org/jira/browse/YARN-7475 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-beta1 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7475.001.patch > > > Container log link is broken -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6483) Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat
[ https://issues.apache.org/jira/browse/YARN-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259564#comment-16259564 ] Juan Rodríguez Hortalá commented on YARN-6483: -- Hi [~asuresh], Thanks a lot for all your comments, I'll send a new patch attending all your suggestions. I'd just need a small clarification, by "update type" you mean adding an additional field with type `RMAppNodeUpdateType` to `NodeReport`? Thanks, Juan > Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes > returned by the Resource Manager as a response to the Application Master > heartbeat > > > Key: YARN-6483 > URL: https://issues.apache.org/jira/browse/YARN-6483 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.1.0 >Reporter: Juan Rodríguez Hortalá >Assignee: Juan Rodríguez Hortalá > Attachments: YARN-6483-v1.patch, YARN-6483.002.patch > > > The DECOMMISSIONING node state is currently used as part of the graceful > decommissioning mechanism to give time for tasks to complete in a node that > is scheduled for decommission, and for reducer tasks to read the shuffle > blocks in that node. Also, YARN effectively blacklists nodes in > DECOMMISSIONING state by assigning them a capacity of 0, to prevent > additional containers to be launched in those nodes, so no more shuffle > blocks are written to the node. This blacklisting is not effective for > applications like Spark, because a Spark executor running in a YARN container > will keep receiving more tasks after the corresponding node has been > blacklisted at the YARN level. We would like to propose a modification of the > YARN heartbeat mechanism so nodes transitioning to DECOMMISSIONING are added > to the list of updated nodes returned by the Resource Manager as a response > to the Application Master heartbeat. This way a Spark application master > would be able to blacklist a DECOMMISSIONING at the Spark level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6483) Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat
[ https://issues.apache.org/jira/browse/YARN-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Rodríguez Hortalá updated YARN-6483: - Attachment: YARN-6483.003.patch > Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes > returned by the Resource Manager as a response to the Application Master > heartbeat > > > Key: YARN-6483 > URL: https://issues.apache.org/jira/browse/YARN-6483 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.1.0 >Reporter: Juan Rodríguez Hortalá >Assignee: Juan Rodríguez Hortalá > Attachments: YARN-6483-v1.patch, YARN-6483.002.patch, > YARN-6483.003.patch > > > The DECOMMISSIONING node state is currently used as part of the graceful > decommissioning mechanism to give time for tasks to complete in a node that > is scheduled for decommission, and for reducer tasks to read the shuffle > blocks in that node. Also, YARN effectively blacklists nodes in > DECOMMISSIONING state by assigning them a capacity of 0, to prevent > additional containers to be launched in those nodes, so no more shuffle > blocks are written to the node. This blacklisting is not effective for > applications like Spark, because a Spark executor running in a YARN container > will keep receiving more tasks after the corresponding node has been > blacklisted at the YARN level. We would like to propose a modification of the > YARN heartbeat mechanism so nodes transitioning to DECOMMISSIONING are added > to the list of updated nodes returned by the Resource Manager as a response > to the Application Master heartbeat. This way a Spark application master > would be able to blacklist a DECOMMISSIONING at the Spark level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6483) Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat
[ https://issues.apache.org/jira/browse/YARN-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261927#comment-16261927 ] Juan Rodríguez Hortalá commented on YARN-6483: -- Hi [~asuresh], I have added a new patch, and also updated the pull request for easy visualization. I have: - provided default implementations for the new methods of NodeReport - reverted the changes to RMNodeDecommissioningEvent to use an Integer again - defined a new enum `org.apache.hadoop.yarn.api.records.NodeUpdateType` that is used as a new optional field for NodeReport, that is only set to a non null value for NodeReports corresponding to node transitions, and it's null for node reports not associated to node transitions like those requested by ClientRMService. I have added assertions for the former case to TestAMRMRPCNodeUpdates, and for the latter to TestClientRMService Thanks again for taking a look! > Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes > returned by the Resource Manager as a response to the Application Master > heartbeat > > > Key: YARN-6483 > URL: https://issues.apache.org/jira/browse/YARN-6483 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.1.0 >Reporter: Juan Rodríguez Hortalá >Assignee: Juan Rodríguez Hortalá > Attachments: YARN-6483-v1.patch, YARN-6483.002.patch, > YARN-6483.003.patch > > > The DECOMMISSIONING node state is currently used as part of the graceful > decommissioning mechanism to give time for tasks to complete in a node that > is scheduled for decommission, and for reducer tasks to read the shuffle > blocks in that node. Also, YARN effectively blacklists nodes in > DECOMMISSIONING state by assigning them a capacity of 0, to prevent > additional containers to be launched in those nodes, so no more shuffle > blocks are written to the node. This blacklisting is not effective for > applications like Spark, because a Spark executor running in a YARN container > will keep receiving more tasks after the corresponding node has been > blacklisted at the YARN level. We would like to propose a modification of the > YARN heartbeat mechanism so nodes transitioning to DECOMMISSIONING are added > to the list of updated nodes returned by the Resource Manager as a response > to the Application Master heartbeat. This way a Spark application master > would be able to blacklist a DECOMMISSIONING at the Spark level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234562#comment-16234562
]
Íñigo Goiri commented on YARN-7276:
---
Hallelujah!
The issue was basically the timeline service dependency in the pom for the unit
tests.
[~subru] do you want to add an addendum adding that dependency to the trunk
patch?
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276-branch-2.007.patch, YARN-7276-branch-2.008.patch,
> YARN-7276-branch-2.009.patch, YARN-7276-branch-2.010.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch, YARN-7276.012.patch,
> YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Fix Version/s: 3.0.0
2.9.0
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 2.9.0, 3.0.0
>
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276-branch-2.007.patch, YARN-7276-branch-2.008.patch,
> YARN-7276-branch-2.009.patch, YARN-7276-branch-2.010.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch, YARN-7276.012.patch,
> YARN-7276.013.patch, YARN-7276.014.patch, YARN-7276.addendum.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276.addendum.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276-branch-2.007.patch, YARN-7276-branch-2.008.patch,
> YARN-7276-branch-2.009.patch, YARN-7276-branch-2.010.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch, YARN-7276.012.patch,
> YARN-7276.013.patch, YARN-7276.014.patch, YARN-7276.addendum.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234698#comment-16234698
]
Íñigo Goiri commented on YARN-7276:
---
Thanks [~subru], committed this way:
* [^YARN-7276.014.patch] into trunk.
* [^YARN-7276-branch-2.010.patch] into branch-2 and branch-2.9.
* [^YARN-7276.addendum.patch] into trunk and branch-3.0 (this branch was cut in
between the addendum and the original patch).
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276-branch-2.007.patch, YARN-7276-branch-2.008.patch,
> YARN-7276-branch-2.009.patch, YARN-7276-branch-2.010.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch, YARN-7276.012.patch,
> YARN-7276.013.patch, YARN-7276.014.patch, YARN-7276.addendum.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234698#comment-16234698
]
Íñigo Goiri edited comment on YARN-7276 at 11/1/17 8:43 PM:
Thanks [~subru], committed this way:
* [^YARN-7276.014.patch] into trunk and branch-3.0.
* [^YARN-7276-branch-2.010.patch] into branch-2 and branch-2.9.
* [^YARN-7276.addendum.patch] into trunk and branch-3.0.
was (Author: elgoiri):
Thanks [~subru], committed this way:
* [^YARN-7276.014.patch] into trunk.
* [^YARN-7276-branch-2.010.patch] into branch-2 and branch-2.9.
* [^YARN-7276.addendum.patch] into trunk and branch-3.0 (this branch was cut in
between the addendum and the original patch).
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 2.9.0, 3.0.0
>
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276-branch-2.007.patch, YARN-7276-branch-2.008.patch,
> YARN-7276-branch-2.009.patch, YARN-7276-branch-2.010.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch, YARN-7276.012.patch,
> YARN-7276.013.patch, YARN-7276.014.patch, YARN-7276.addendum.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7383) Node resource not parsed correctly for resource names containing dot
[
https://issues.apache.org/jira/browse/YARN-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák reassigned YARN-7383:
---
Assignee: Gergely Novák
> Node resource not parsed correctly for resource names containing dot
>
>
> Key: YARN-7383
> URL: https://issues.apache.org/jira/browse/YARN-7383
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Jonathan Hung
>Assignee: Gergely Novák
>Priority: Major
>
> Currently when getting resource capabilities in {{node-resources.xml}}, the
> following parsing code is used: {noformat}String[] parts =
> prop.split("\\.");
> LOG.info("Found resource entry " + prop);
> if (parts.length == 4) {
> String resourceType = parts[3];
> if (!nodeResources.containsKey(resourceType)) {
> nodeResources
> .put(resourceType, ResourceInformation.newInstance(resourceType));
> }
> String units = getUnits(value);
> Long resourceValue =
> Long.valueOf(value.substring(0, value.length() - units.length()));
> nodeResources.get(resourceType).setValue(resourceValue);
> nodeResources.get(resourceType).setUnits(units);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Setting value for resource type " + resourceType + " to "
> + resourceValue + " with units " + units);
> }
> }{noformat}
> If resource names contain dots though (e.g. for resource names in a
> namespace) this will not parse correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7383) Node resource not parsed correctly for resource names containing dot
[
https://issues.apache.org/jira/browse/YARN-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-7383:
Attachment: YARN-7383.001.patch
I updated a patch for this problem, could you please take a look at it?
> Node resource not parsed correctly for resource names containing dot
>
>
> Key: YARN-7383
> URL: https://issues.apache.org/jira/browse/YARN-7383
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Jonathan Hung
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7383.001.patch
>
>
> Currently when getting resource capabilities in {{node-resources.xml}}, the
> following parsing code is used: {noformat}String[] parts =
> prop.split("\\.");
> LOG.info("Found resource entry " + prop);
> if (parts.length == 4) {
> String resourceType = parts[3];
> if (!nodeResources.containsKey(resourceType)) {
> nodeResources
> .put(resourceType, ResourceInformation.newInstance(resourceType));
> }
> String units = getUnits(value);
> Long resourceValue =
> Long.valueOf(value.substring(0, value.length() - units.length()));
> nodeResources.get(resourceType).setValue(resourceValue);
> nodeResources.get(resourceType).setUnits(units);
> if (LOG.isDebugEnabled()) {
> LOG.debug("Setting value for resource type " + resourceType + " to "
> + resourceValue + " with units " + units);
> }
> }{noformat}
> If resource names contain dots though (e.g. for resource names in a
> namespace) this will not parse correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236155#comment-16236155
]
Gergely Novák commented on YARN-7370:
-
Sorry, I already did (patch #5).
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7370.001.patch, YARN-7370.002.patch,
> YARN-7370.003.patch, YARN-7370.004.patch, YARN-7370.005.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-7370:
Attachment: YARN-7370.005.patch
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7370.001.patch, YARN-7370.002.patch,
> YARN-7370.003.patch, YARN-7370.004.patch, YARN-7370.005.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7434) Router getApps REST invocation fails with multiple RMs
[ https://issues.apache.org/jira/browse/YARN-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236872#comment-16236872 ] Íñigo Goiri commented on YARN-7434: --- For testing this, we need a federated mini YARN cluster with a couple Resource Managers. > Router getApps REST invocation fails with multiple RMs > -- > > Key: YARN-7434 > URL: https://issues.apache.org/jira/browse/YARN-7434 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Subru Krishnan >Assignee: Íñigo Goiri >Priority: Critical > Attachments: YARN-7434.000.patch > > > Router uses threads to invoke getApps in parallel with multiple RMs and has a > concurrency bug caused by sharing of the HTTP request object. This jira > tracks the changes to fix the multi-threading issue by cloning the request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7434) Router getApps REST invocation fails with multiple RMs
[ https://issues.apache.org/jira/browse/YARN-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-7434: -- Attachment: YARN-7434.000.patch > Router getApps REST invocation fails with multiple RMs > -- > > Key: YARN-7434 > URL: https://issues.apache.org/jira/browse/YARN-7434 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Subru Krishnan >Assignee: Íñigo Goiri >Priority: Critical > Attachments: YARN-7434.000.patch > > > Router uses threads to invoke getApps in parallel with multiple RMs and has a > concurrency bug caused by sharing of the HTTP request object. This jira > tracks the changes to fix the multi-threading issue by cloning the request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7434) Router getApps REST invocation fails with multiple RMs
[ https://issues.apache.org/jira/browse/YARN-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236911#comment-16236911 ] Íñigo Goiri commented on YARN-7434: --- This one patch should apply to both branches. The issue might be that the supresswarning might give issues with Java 8. If that's the case, we will need a different patch for trunk. > Router getApps REST invocation fails with multiple RMs > -- > > Key: YARN-7434 > URL: https://issues.apache.org/jira/browse/YARN-7434 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Subru Krishnan >Assignee: Íñigo Goiri >Priority: Critical > Attachments: YARN-7434.000.patch > > > Router uses threads to invoke getApps in parallel with multiple RMs and has a > concurrency bug caused by sharing of the HTTP request object. This jira > tracks the changes to fix the multi-threading issue by cloning the request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7434) Router getApps REST invocation fails with multiple RMs
[
https://issues.apache.org/jira/browse/YARN-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7434:
--
Attachment: YARN-7434.001.patch
[~subru], moving the {{clone()}} outside works.
Added patch with that.
> Router getApps REST invocation fails with multiple RMs
> --
>
> Key: YARN-7434
> URL: https://issues.apache.org/jira/browse/YARN-7434
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Íñigo Goiri
>Priority: Critical
> Attachments: YARN-7434.000.patch, YARN-7434.001.patch
>
>
> Router uses threads to invoke getApps in parallel with multiple RMs and has a
> concurrency bug caused by sharing of the HTTP request object. This jira
> tracks the changes to fix the multi-threading issue by cloning the request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7435) Add MiniYARNCluster for Federation
[
https://issues.apache.org/jira/browse/YARN-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236929#comment-16236929
]
Íñigo Goiri commented on YARN-7435:
---
This should be used in {{TestFederationInterceptorREST}}.
> Add MiniYARNCluster for Federation
> --
>
> Key: YARN-7435
> URL: https://issues.apache.org/jira/browse/YARN-7435
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Íñigo Goiri
>Priority: Normal
>
> Currently the unit tests for the Router use {{JavaProcess}} to start NMs,
> RMs, and Routers. All this should be encapsulated into an extension of the
> MiniYARNCluster which sets up a State Store.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7435) Add MiniYARNCluster for Federation
Íñigo Goiri created YARN-7435:
-
Summary: Add MiniYARNCluster for Federation
Key: YARN-7435
URL: https://issues.apache.org/jira/browse/YARN-7435
Project: Hadoop YARN
Issue Type: Test
Reporter: Íñigo Goiri
Priority: Normal
Currently the unit tests for the Router use {{JavaProcess}} to start NMs, RMs,
and Routers. All this should be encapsulated into an extension of the
MiniYARNCluster which sets up a State Store.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7434) Router getApps REST invocation fails with multiple RMs
[ https://issues.apache.org/jira/browse/YARN-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236932#comment-16236932 ] Íñigo Goiri commented on YARN-7434: --- I added YARN-7435 to track the addition of end to end tests. > Router getApps REST invocation fails with multiple RMs > -- > > Key: YARN-7434 > URL: https://issues.apache.org/jira/browse/YARN-7434 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Subru Krishnan >Assignee: Íñigo Goiri >Priority: Critical > Attachments: YARN-7434.000.patch, YARN-7434.001.patch > > > Router uses threads to invoke getApps in parallel with multiple RMs and has a > concurrency bug caused by sharing of the HTTP request object. This jira > tracks the changes to fix the multi-threading issue by cloning the request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6907) Node information page in the old web UI should report resource types
[ https://issues.apache.org/jira/browse/YARN-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák reassigned YARN-6907: --- Assignee: Gergely Novák > Node information page in the old web UI should report resource types > > > Key: YARN-6907 > URL: https://issues.apache.org/jira/browse/YARN-6907 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Gergely Novák > Labels: newbie > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6907) Node information page in the old web UI should report resource types
[ https://issues.apache.org/jira/browse/YARN-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-6907: Attachment: YARN-6907.001.patch > Node information page in the old web UI should report resource types > > > Key: YARN-6907 > URL: https://issues.apache.org/jira/browse/YARN-6907 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Gergely Novák > Labels: newbie > Attachments: YARN-6907.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7453) RM fail to switch to active after first successful start
[
https://issues.apache.org/jira/browse/YARN-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242805#comment-16242805
]
Íñigo Goiri commented on YARN-7453:
---
>From the vote for 3.0.0-beta1 RC0, I can see [~rohithsharma] tested HA.
[Link to the
archive|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201710.mbox/ajax/%3CCAGi%3DjZDPP%2BUwamnPQz89t4xy2bnPbvDUtYRRWG6sN3Dnojf05w%40mail.gmail.com%3E]:
{code}
Built from source and deployed YARN HA cluster with ATSv2 enabled in
non-secured cluster.
- tested for RM HA/work-preservring-restart/ NM-work-preserving restart for
ATSv2 entities.
- verified all ATSv2 REST end points to retrieve the entities
- ran sample MR jobs and distributed jobs
{code}
If I'm not wrong, YARN-7053 and HADOOP-14741 were already in 3.0.0-beta1.
> RM fail to switch to active after first successful start
>
>
> Key: YARN-7453
> URL: https://issues.apache.org/jira/browse/YARN-7453
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0, 3.1.0
>Reporter: Rohith Sharma K S
>Priority: Blocker
>
> It is observed that RM fail to switch to ACTIVE after first successful start!
> The below exception throws when RM is switching from ACTIVE->STANDBY->ACTIVE.
> This continues in loop!
> {noformat}
> 2017-11-07 15:08:11,664 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning
> to active state
> 2017-11-07 15:08:11,669 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Recovery
> started
> 2017-11-07 15:08:11,669 INFO
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Loaded
> RM state version info 1.5
> 2017-11-07 15:08:11,670 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Failed to
> load/recover state
> org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1006)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:910)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl.doOperation(CuratorTransactionImpl.java:159)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl.access$200(CuratorTransactionImpl.java:44)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:129)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl$2.call(CuratorTransactionImpl.java:125)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
> at
> org.apache.curator.framework.imps.CuratorTransactionImpl.commit(CuratorTransactionImpl.java:122)
> at
> org.apache.hadoop.util.curator.ZKCuratorManager$SafeTransaction.commit(ZKCuratorManager.java:403)
> at
> org.apache.hadoop.util.curator.ZKCuratorManager.safeSetData(ZKCuratorManager.java:372)
> at
> org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.getAndIncrementEpoch(ZKRMStateStore.java:493)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:771)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1162)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1202)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1198)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1198)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:320)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:894)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:473)
> at
&g
[jira] [Commented] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235613#comment-16235613
]
Gergely Novák commented on YARN-7370:
-
Added the above 4 properties to the log statement in patch #3.
As for the float-double change, if you check the
{{ProportionalCapacityPreemptionPolicy}} code, both of the modified default
properties are used with getDouble() methods, meaning that they are
automatically converted to double (and neither
{{DEFAULT_PREEMPTION_MAX_IGNORED_OVER_CAPACITY}} nor
{{DEFAULT_PREEMPTION_NATURAL_TERMINATION_FACTOR}} is used anywhere else in the
codebase) - so I didn't actually change the type that is actually used (the
type of {{maxIgnoredOverCapacity}} and {{naturalTerminationFactor}} was and
still is double). The reason I changed the default constants is the precision
problem of the automatic conversion: 0.1 becomes 0.1000149011612 and it
looked funny in the newly introduced log message.
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7370.001.patch, YARN-7370.002.patch,
> YARN-7370.003.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-7370:
Attachment: YARN-7370.003.patch
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7370.001.patch, YARN-7370.002.patch,
> YARN-7370.003.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.010.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276-branch-2.007.patch, YARN-7276-branch-2.008.patch,
> YARN-7276-branch-2.009.patch, YARN-7276-branch-2.010.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch, YARN-7276.012.patch,
> YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: (was: YARN-7276-branch-2.010.patch)
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276-branch-2.007.patch, YARN-7276-branch-2.008.patch,
> YARN-7276-branch-2.009.patch, YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch,
> YARN-7276.005.patch, YARN-7276.006.patch, YARN-7276.007.patch,
> YARN-7276.009.patch, YARN-7276.010.patch, YARN-7276.011.patch,
> YARN-7276.012.patch, YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-7370:
Attachment: YARN-7370.004.patch
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7370.001.patch, YARN-7370.002.patch,
> YARN-7370.003.patch, YARN-7370.004.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235880#comment-16235880
]
Gergely Novák commented on YARN-7370:
-
Thanks [~sunilg] for the suggestion, I removed the null check in patch #4, now
we log the properties during the serviceInit, too.
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-7370.001.patch, YARN-7370.002.patch,
> YARN-7370.003.patch, YARN-7370.004.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7434) Router getApps REST invocation fails with multiple RMs
[ https://issues.apache.org/jira/browse/YARN-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-7434: -- Fix Version/s: 3.0.0 2.9.0 > Router getApps REST invocation fails with multiple RMs > -- > > Key: YARN-7434 > URL: https://issues.apache.org/jira/browse/YARN-7434 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Subru Krishnan >Assignee: Íñigo Goiri >Priority: Critical > Fix For: 2.9.0, 3.0.0 > > Attachments: YARN-7434.000.patch, YARN-7434.001.patch > > > Router uses threads to invoke getApps in parallel with multiple RMs and has a > concurrency bug caused by sharing of the HTTP request object. This jira > tracks the changes to fix the multi-threading issue by cloning the request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7434) Router getApps REST invocation fails with multiple RMs
[ https://issues.apache.org/jira/browse/YARN-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16237072#comment-16237072 ] Íñigo Goiri commented on YARN-7434: --- YETUS only complained about the tests which we are tracking in YARN-7435 as this is a release blocker. Committing to trunk and cherry picking to branch-3.0, branch-2 and branch-2.9. > Router getApps REST invocation fails with multiple RMs > -- > > Key: YARN-7434 > URL: https://issues.apache.org/jira/browse/YARN-7434 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Subru Krishnan >Assignee: Íñigo Goiri >Priority: Critical > Attachments: YARN-7434.000.patch, YARN-7434.001.patch > > > Router uses threads to invoke getApps in parallel with multiple RMs and has a > concurrency bug caused by sharing of the HTTP request object. This jira > tracks the changes to fix the multi-threading issue by cloning the request. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7608) Incorrect sTarget column causing DataTable warning on RM application and scheduler web page
[
https://issues.apache.org/jira/browse/YARN-7608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283222#comment-16283222
]
Gergely Novák commented on YARN-7608:
-
Nice catch, patch #2 fixes the fair scheduler page, too.
What about the else branch? The only usage of the method where both
{{isFairSchedulerPage}} and {{isResourceManager}} is false is
{{AHSView::preHead()}}, but as far as I could see, all the subclasses of
{{AHSView}} overrides {{preHead}}, so eventually it's not used anywhere..
> Incorrect sTarget column causing DataTable warning on RM application and
> scheduler web page
> ---
>
> Key: YARN-7608
> URL: https://issues.apache.org/jira/browse/YARN-7608
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, webapp
>Affects Versions: 2.9.0
>Reporter: Weiwei Yang
>Assignee: Gergely Novák
> Attachments: YARN-7608.001.patch, YARN-7608.002.patch
>
>
> On a cluster built from latest trunk, click {{% of Queue}} gives following
> warning
> {noformat}
> DataTable warning (tableID="'apps'): Requested unknown parameter '15' from
> the data source for row 0
> {noformat}
> {{% of Cluster}} doesn't have this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7608) Incorrect sTarget column causing DataTable warning on RM application and scheduler web page
[
https://issues.apache.org/jira/browse/YARN-7608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-7608:
Attachment: YARN-7608.002.patch
> Incorrect sTarget column causing DataTable warning on RM application and
> scheduler web page
> ---
>
> Key: YARN-7608
> URL: https://issues.apache.org/jira/browse/YARN-7608
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, webapp
>Affects Versions: 2.9.0
>Reporter: Weiwei Yang
>Assignee: Gergely Novák
> Attachments: YARN-7608.001.patch, YARN-7608.002.patch
>
>
> On a cluster built from latest trunk, click {{% of Queue}} gives following
> warning
> {noformat}
> DataTable warning (tableID="'apps'): Requested unknown parameter '15' from
> the data source for row 0
> {noformat}
> {{% of Cluster}} doesn't have this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7594) TestNMWebServices.testGetNMResourceInfo fails on trunk
Gergely Novák created YARN-7594:
---
Summary: TestNMWebServices.testGetNMResourceInfo fails on trunk
Key: YARN-7594
URL: https://issues.apache.org/jira/browse/YARN-7594
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager, webapp
Reporter: Gergely Novák
{code}
// Access resource-2 should fail (null NMResourceInfo returned).
JSONObject json = response.getEntity(JSONObject.class);
assertIncludesException(json);
{code}
fails with
{noformat}
java.lang.AssertionError ...
at
org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices.assertIncludesException(TestNMWebServices.java:541)
at
org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices.testGetNMResourceInfo(TestNMWebServices.java:461)...
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7594) TestNMWebServices.testGetNMResourceInfo fails on trunk
[
https://issues.apache.org/jira/browse/YARN-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák reassigned YARN-7594:
---
Assignee: Gergely Novák
> TestNMWebServices.testGetNMResourceInfo fails on trunk
> --
>
> Key: YARN-7594
> URL: https://issues.apache.org/jira/browse/YARN-7594
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager, webapp
>Reporter: Gergely Novák
>Assignee: Gergely Novák
> Attachments: YARN-7594.001.patch
>
>
> {code}
> // Access resource-2 should fail (null NMResourceInfo returned).
> JSONObject json = response.getEntity(JSONObject.class);
> assertIncludesException(json);
> {code}
> fails with
> {noformat}
> java.lang.AssertionError ...
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices.assertIncludesException(TestNMWebServices.java:541)
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices.testGetNMResourceInfo(TestNMWebServices.java:461)...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7594) TestNMWebServices.testGetNMResourceInfo fails on trunk
[
https://issues.apache.org/jira/browse/YARN-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-7594:
Attachment: YARN-7594.001.patch
> TestNMWebServices.testGetNMResourceInfo fails on trunk
> --
>
> Key: YARN-7594
> URL: https://issues.apache.org/jira/browse/YARN-7594
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager, webapp
>Reporter: Gergely Novák
> Attachments: YARN-7594.001.patch
>
>
> {code}
> // Access resource-2 should fail (null NMResourceInfo returned).
> JSONObject json = response.getEntity(JSONObject.class);
> assertIncludesException(json);
> {code}
> fails with
> {noformat}
> java.lang.AssertionError ...
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices.assertIncludesException(TestNMWebServices.java:541)
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices.testGetNMResourceInfo(TestNMWebServices.java:461)...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6907) Node information page in the old web UI should report resource types
[ https://issues.apache.org/jira/browse/YARN-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-6907: Attachment: YARN-6907.003.patch Patch #3 fixes the affected unit tests. > Node information page in the old web UI should report resource types > > > Key: YARN-6907 > URL: https://issues.apache.org/jira/browse/YARN-6907 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 3.0.0, 3.1.0 >Reporter: Daniel Templeton >Assignee: Gergely Novák > Labels: newbie > Attachments: YARN-6907.001.patch, YARN-6907.002.patch, > YARN-6907.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6907) Node information page in the old web UI should report resource types
[
https://issues.apache.org/jira/browse/YARN-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274418#comment-16274418
]
Gergely Novák commented on YARN-6907:
-
{{TestNMWebServices.testGetNMResourceInfo}} fails without the patch, too.
Should I open a ticket for it?
> Node information page in the old web UI should report resource types
>
>
> Key: YARN-6907
> URL: https://issues.apache.org/jira/browse/YARN-6907
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Daniel Templeton
>Assignee: Gergely Novák
> Labels: newbie
> Attachments: YARN-6907.001.patch, YARN-6907.002.patch,
> YARN-6907.003.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7594) TestNMWebServices.testGetNMResourceInfo fails on trunk
[
https://issues.apache.org/jira/browse/YARN-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274440#comment-16274440
]
Gergely Novák commented on YARN-7594:
-
Since YARN-7573 {{getNMResourceInfo()}} doesn't throw Exception for wrong
resource name, but returns an emtpy JSON.
> TestNMWebServices.testGetNMResourceInfo fails on trunk
> --
>
> Key: YARN-7594
> URL: https://issues.apache.org/jira/browse/YARN-7594
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager, webapp
>Reporter: Gergely Novák
>
> {code}
> // Access resource-2 should fail (null NMResourceInfo returned).
> JSONObject json = response.getEntity(JSONObject.class);
> assertIncludesException(json);
> {code}
> fails with
> {noformat}
> java.lang.AssertionError ...
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices.assertIncludesException(TestNMWebServices.java:541)
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServices.testGetNMResourceInfo(TestNMWebServices.java:461)...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7608) RM UI prompted DataTable warning while clicking on percent of queue column
[
https://issues.apache.org/jira/browse/YARN-7608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák reassigned YARN-7608:
---
Assignee: Gergely Novák
> RM UI prompted DataTable warning while clicking on percent of queue column
> --
>
> Key: YARN-7608
> URL: https://issues.apache.org/jira/browse/YARN-7608
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, webapp
>Affects Versions: 2.9.0
>Reporter: Weiwei Yang
>Assignee: Gergely Novák
> Attachments: YARN-7608.001.patch
>
>
> On a cluster built from latest trunk, click {{% of Queue}} gives following
> warning
> {noformat}
> DataTable warning (tableID="'apps'): Requested unknown parameter '15' from
> the data source for row 0
> {noformat}
> {{% of Cluster}} doesn't have this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7608) RM UI prompted DataTable warning while clicking on percent of queue column
[
https://issues.apache.org/jira/browse/YARN-7608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281939#comment-16281939
]
Gergely Novák commented on YARN-7608:
-
This is caused by YARN-6752 which added two more columns to the table, thus
changing the index of the progress column from 15 to 17. The above patch solves
this issue.
> RM UI prompted DataTable warning while clicking on percent of queue column
> --
>
> Key: YARN-7608
> URL: https://issues.apache.org/jira/browse/YARN-7608
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, webapp
>Affects Versions: 2.9.0
>Reporter: Weiwei Yang
> Attachments: YARN-7608.001.patch
>
>
> On a cluster built from latest trunk, click {{% of Queue}} gives following
> warning
> {noformat}
> DataTable warning (tableID="'apps'): Requested unknown parameter '15' from
> the data source for row 0
> {noformat}
> {{% of Cluster}} doesn't have this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7608) RM UI prompted DataTable warning while clicking on percent of queue column
[
https://issues.apache.org/jira/browse/YARN-7608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-7608:
Attachment: YARN-7608.001.patch
> RM UI prompted DataTable warning while clicking on percent of queue column
> --
>
> Key: YARN-7608
> URL: https://issues.apache.org/jira/browse/YARN-7608
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, webapp
>Affects Versions: 2.9.0
>Reporter: Weiwei Yang
>Assignee: Gergely Novák
> Attachments: YARN-7608.001.patch
>
>
> On a cluster built from latest trunk, click {{% of Queue}} gives following
> warning
> {noformat}
> DataTable warning (tableID="'apps'): Requested unknown parameter '15' from
> the data source for row 0
> {noformat}
> {{% of Cluster}} doesn't have this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7242) Support specify values of different resource types in DistributedShell for easier testing
[ https://issues.apache.org/jira/browse/YARN-7242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280734#comment-16280734 ] Gergely Novák commented on YARN-7242: - [~leftnoteasy] I'd love to work on this issue. The 2nd patch fixes the checkstyle issues and the broken unit test, however, it doesn't contain the new format you requested, because I have some questions regarding this: YARN-5881 is not yet merged to trunk, so I cannot use any of that code. Should I create a custom util in the DistributedShell app that converts the above format to a Resource class? In YARN-5881 this logic is implemented in CapacitySchedulerConfiguration, I feel that the same logic should be extracted to a common class in order to eliminate any duplications, e.g. to Resource or ResourceUtils. Especially that the handling of the "memory" -> "memory-mb" conversion is not trivial and that the format should support expansion (new resource types and formats). > Support specify values of different resource types in DistributedShell for > easier testing > - > > Key: YARN-7242 > URL: https://issues.apache.org/jira/browse/YARN-7242 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Gergely Novák >Priority: Critical > Labels: newbie > Attachments: YARN-7242.001.patch > > > Currently, DS supports specify resource profile, it's better to allow user to > directly specify resource keys/values from command line. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7242) Support specify values of different resource types in DistributedShell for easier testing
[ https://issues.apache.org/jira/browse/YARN-7242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-7242: Attachment: YARN-7242.002.patch > Support specify values of different resource types in DistributedShell for > easier testing > - > > Key: YARN-7242 > URL: https://issues.apache.org/jira/browse/YARN-7242 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Gergely Novák >Priority: Critical > Labels: newbie > Attachments: YARN-7242.001.patch, YARN-7242.002.patch > > > Currently, DS supports specify resource profile, it's better to allow user to > directly specify resource keys/values from command line. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-7370:
Attachment: YARN-7370.002.patch
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
> Attachments: YARN-7370.001.patch, YARN-7370.002.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224824#comment-16224824
]
Gergely Novák commented on YARN-7370:
-
[~eepayne] Thank you for your suggestions, addressed them in patch #2.
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
> Attachments: YARN-7370.001.patch, YARN-7370.002.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.005.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.005.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch, YARN-7276.012.patch,
> YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: (was: YARN-7276.008.patch)
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276.011.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219621#comment-16219621
]
Íñigo Goiri commented on YARN-7276:
---
Thanks for the comments [~giovanni.fumarola], I uploaded 011 tackling them.
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-7370:
Attachment: YARN-7370.001.patch
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
> Attachments: YARN-7370.001.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276.009.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219035#comment-16219035
]
Gergely Novák commented on YARN-7370:
-
[~eepayne] He must have meant YARN-6124, not YARN-6142.
I attached a first WiP version for review.
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
> Attachments: YARN-7370.001.patch
>
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276.010.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3661) Basic Federation UI
[ https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-3661: -- Attachment: YARN-3661-branch-2.002.patch Re-uploaded 002 for branch-2 to trigger jenkins. > Basic Federation UI > > > Key: YARN-3661 > URL: https://issues.apache.org/jira/browse/YARN-3661 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Íñigo Goiri > Attachments: YARN-3661-000.patch, YARN-3661-001.patch, > YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch, > YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch, > YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch, > YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch, > YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch, > YARN-3661-branch-2.002.patch > > > The UIs provided by each RM, provide a correct "local" view of what is > running in a sub-cluster. In the context of federation we need new > UIs that can track load, jobs, users across sub-clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3661) Basic Federation UI
[ https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-3661: -- Attachment: (was: YARN-3661-branch-2.002.patch) > Basic Federation UI > > > Key: YARN-3661 > URL: https://issues.apache.org/jira/browse/YARN-3661 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Íñigo Goiri > Attachments: YARN-3661-000.patch, YARN-3661-001.patch, > YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch, > YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch, > YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch, > YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch, > YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch, > YARN-3661-branch-2.002.patch > > > The UIs provided by each RM, provide a correct "local" view of what is > running in a sub-cluster. In the context of federation we need new > UIs that can track load, jobs, users across sub-clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.003.patch
Java 7 has a warning for:
{code}
public Map<String, String[]> getParameterMap() {
return hsr.getParameterMap();
}
{code}
This doesn't happen in trunk (Java 8), and adding a {{@SuppressWarnings}}
triggers an unused reference. So I'm diverging there.
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch,
> YARN-7276.005.patch, YARN-7276.006.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3661) Basic Federation UI
[ https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-3661: -- Fix Version/s: 3.0 2.9.0 > Basic Federation UI > > > Key: YARN-3661 > URL: https://issues.apache.org/jira/browse/YARN-3661 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Íñigo Goiri > Fix For: 2.9.0, 3.0 > > Attachments: YARN-3661-000.patch, YARN-3661-001.patch, > YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch, > YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch, > YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch, > YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch, > YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch, > YARN-3661-branch-2.002.patch > > > The UIs provided by each RM, provide a correct "local" view of what is > running in a sub-cluster. In the context of federation we need new > UIs that can track load, jobs, users across sub-clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3661) Basic Federation UI
[ https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217689#comment-16217689 ] Íñigo Goiri commented on YARN-3661: --- Thanks for the review [~curino]. Tested locally and it works. Committed this and left the unit test issue for YARN-7276. > Basic Federation UI > > > Key: YARN-3661 > URL: https://issues.apache.org/jira/browse/YARN-3661 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Íñigo Goiri > Attachments: YARN-3661-000.patch, YARN-3661-001.patch, > YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch, > YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch, > YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch, > YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch, > YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch, > YARN-3661-branch-2.002.patch > > > The UIs provided by each RM, provide a correct "local" view of what is > running in a sub-cluster. In the context of federation we need new > UIs that can track load, jobs, users across sub-clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3661) Basic Federation UI
[
https://issues.apache.org/jira/browse/YARN-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217672#comment-16217672
]
Íñigo Goiri commented on YARN-3661:
---
The problem is that {{TestRouterWebServicesREST}} cannot start and crashes
without cleaning.
I think we should commit this one and add the fix to this problem in YARN-7276.
> Basic Federation UI
>
>
> Key: YARN-3661
> URL: https://issues.apache.org/jira/browse/YARN-3661
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Íñigo Goiri
> Attachments: YARN-3661-000.patch, YARN-3661-001.patch,
> YARN-3661-002.patch, YARN-3661-003.patch, YARN-3661-004.patch,
> YARN-3661-005.patch, YARN-3661-006.patch, YARN-3661-007.patch,
> YARN-3661-008.patch, YARN-3661-009.patch, YARN-3661-010.patch,
> YARN-3661-011.patch, YARN-3661-012.patch, YARN-3661-013.patch,
> YARN-3661-014.patch, YARN-3661-015.patch, YARN-3661-branch-2.001.patch,
> YARN-3661-branch-2.002.patch
>
>
> The UIs provided by each RM, provide a correct "local" view of what is
> running in a sub-cluster. In the context of federation we need new
> UIs that can track load, jobs, users across sub-clusters.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.004.patch
YARN-7190 in branch-2 tuned the way the classpath was setup. I had to add the
classpath to the start of the RM.
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276.007.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7370) Preemption properties should be refreshable
[
https://issues.apache.org/jira/browse/YARN-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák reassigned YARN-7370:
---
Assignee: Gergely Novák
> Preemption properties should be refreshable
> ---
>
> Key: YARN-7370
> URL: https://issues.apache.org/jira/browse/YARN-7370
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0, 3.0.0-alpha3
>Reporter: Eric Payne
>Assignee: Gergely Novák
>
> At least the properties for {{max-allowable-limit}} and {{minimum-threshold}}
> should be refreshable. It would also be nice to make
> {{intra-queue-preemption.enabled}} and {{preemption-order-policy}}
> refreshable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276.008.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.008.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276.004.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276.003.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276.005.patch
Thanks [~subru] for the comments.
I added the multithreaded test in 005.
The rest were already done in 004.
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch,
> YARN-7276.005.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.000.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch, YARN-7276.000.patch,
> YARN-7276.001.patch, YARN-7276.002.patch, YARN-7276.003.patch,
> YARN-7276.004.patch, YARN-7276.005.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276.006.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.001.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch,
> YARN-7276.005.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.002.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.007.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276-branch-2.007.patch, YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch,
> YARN-7276.005.patch, YARN-7276.006.patch, YARN-7276.007.patch,
> YARN-7276.009.patch, YARN-7276.010.patch, YARN-7276.011.patch,
> YARN-7276.012.patch, YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.008.patch
Trying to debug the issue with the classpath.
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276-branch-2.007.patch, YARN-7276-branch-2.008.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch, YARN-7276.012.patch,
> YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.009.patch
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276-branch-2.007.patch, YARN-7276-branch-2.008.patch,
> YARN-7276-branch-2.009.patch, YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch,
> YARN-7276.005.patch, YARN-7276.006.patch, YARN-7276.007.patch,
> YARN-7276.009.patch, YARN-7276.010.patch, YARN-7276.011.patch,
> YARN-7276.012.patch, YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.010.patch
Adding the timeline service as a dependency.
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276-branch-2.007.patch, YARN-7276-branch-2.008.patch,
> YARN-7276-branch-2.009.patch, YARN-7276-branch-2.010.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch, YARN-7276.012.patch,
> YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: (was: YARN-7276-branch-2.005.patch)
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch,
> YARN-7276.005.patch, YARN-7276.006.patch, YARN-7276.007.patch,
> YARN-7276.009.patch, YARN-7276.010.patch, YARN-7276.011.patch,
> YARN-7276.012.patch, YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.005.patch
Not sure the warnings are correct (i.e., checkstyle), I'm uploading
[^YARN-7276-branch-2.005.patch] again to see if it goes away.
One error that I've seen before is about the unchecked conversion with:
{code}
public Map<String, String[]> getParameterMap() {
return hsr.getParameterMap();
}
{code}
Trunk (Java 8) didn't complain but with branch-2 is happening once in a while.
Not sure how to fix it.
I found the following:
https://stackoverflow.com/questions/5317443/java-unchecked-conversion
However, it doesn't look like a clean solution.
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch,
> YARN-7276.005.patch, YARN-7276.006.patch, YARN-7276.007.patch,
> YARN-7276.009.patch, YARN-7276.010.patch, YARN-7276.011.patch,
> YARN-7276.012.patch, YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: YARN-7276-branch-2.006.patch
Fixed javac issue with explicit cast.
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276-branch-2.006.patch,
> YARN-7276.000.patch, YARN-7276.001.patch, YARN-7276.002.patch,
> YARN-7276.003.patch, YARN-7276.004.patch, YARN-7276.005.patch,
> YARN-7276.006.patch, YARN-7276.007.patch, YARN-7276.009.patch,
> YARN-7276.010.patch, YARN-7276.011.patch, YARN-7276.012.patch,
> YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7276) Federation Router Web Service fixes
[
https://issues.apache.org/jira/browse/YARN-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-7276:
--
Attachment: (was: YARN-7276-branch-2.005.patch)
> Federation Router Web Service fixes
> ---
>
> Key: YARN-7276
> URL: https://issues.apache.org/jira/browse/YARN-7276
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Attachments: YARN-7276-branch-2.000.patch,
> YARN-7276-branch-2.001.patch, YARN-7276-branch-2.002.patch,
> YARN-7276-branch-2.003.patch, YARN-7276-branch-2.004.patch,
> YARN-7276-branch-2.005.patch, YARN-7276.000.patch, YARN-7276.001.patch,
> YARN-7276.002.patch, YARN-7276.003.patch, YARN-7276.004.patch,
> YARN-7276.005.patch, YARN-7276.006.patch, YARN-7276.007.patch,
> YARN-7276.009.patch, YARN-7276.010.patch, YARN-7276.011.patch,
> YARN-7276.012.patch, YARN-7276.013.patch, YARN-7276.014.patch
>
>
> While testing YARN-3661, I found a few issues with the REST interface in the
> Router:
> * No support for empty content (error 204)
> * Media type support
> * Attributes in {{FederationInterceptorREST}}
> * Support for empty states and labels
> * DefaultMetricsSystem initialization is missing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7675) The new UI won't load for pre 2.8 Hadoop versions because queueCapacitiesByPartition is missing from the scheduler API
[
https://issues.apache.org/jira/browse/YARN-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák reassigned YARN-7675:
---
Assignee: Gergely Novák
> The new UI won't load for pre 2.8 Hadoop versions because
> queueCapacitiesByPartition is missing from the scheduler API
> --
>
> Key: YARN-7675
> URL: https://issues.apache.org/jira/browse/YARN-7675
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Gergely Novák
>Assignee: Gergely Novák
> Attachments: YARN-7675.001.patch
>
>
> If we connect the new YARN UI to any Hadoop versions older than 2.8 it won't
> load. The console shows this trace:
> {noformat}
> TypeError: Cannot read property 'queueCapacitiesByPartition' of undefined
> at Class.normalizeSingleResponse (yarn-ui.js:13903)
> at Class.superWrapper [as normalizeSingleResponse] (vendor.js:31811)
> at Class.handleQueue (yarn-ui.js:13928)
> at Class.normalizeArrayResponse (yarn-ui.js:13952)
> at Class.normalizeQueryResponse (vendor.js:101566)
> at Class.normalizeResponse (vendor.js:101468)
> at
> ember$data$lib$system$store$serializer$response$$normalizeResponseHelper
> (vendor.js:95345)
> at vendor.js:95672
> at Backburner.run (vendor.js:10426)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7675) The new UI won't load for pre 2.8 Hadoop versions because queueCapacitiesByPartition is missing from the scheduler API
[
https://issues.apache.org/jira/browse/YARN-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-7675:
Attachment: YARN-7675.001.patch
> The new UI won't load for pre 2.8 Hadoop versions because
> queueCapacitiesByPartition is missing from the scheduler API
> --
>
> Key: YARN-7675
> URL: https://issues.apache.org/jira/browse/YARN-7675
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Gergely Novák
>Assignee: Gergely Novák
> Attachments: YARN-7675.001.patch
>
>
> If we connect the new YARN UI to any Hadoop versions older than 2.8 it won't
> load. The console shows this trace:
> {noformat}
> TypeError: Cannot read property 'queueCapacitiesByPartition' of undefined
> at Class.normalizeSingleResponse (yarn-ui.js:13903)
> at Class.superWrapper [as normalizeSingleResponse] (vendor.js:31811)
> at Class.handleQueue (yarn-ui.js:13928)
> at Class.normalizeArrayResponse (yarn-ui.js:13952)
> at Class.normalizeQueryResponse (vendor.js:101566)
> at Class.normalizeResponse (vendor.js:101468)
> at
> ember$data$lib$system$store$serializer$response$$normalizeResponseHelper
> (vendor.js:95345)
> at vendor.js:95672
> at Backburner.run (vendor.js:10426)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[ https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464438#comment-16464438 ] Íñigo Goiri commented on YARN-8163: --- +1 on [^YARN-8163.004.patch]. Committing to trunk. Thanks [~abmodi] for the patch and [~giovanni.fumarola] for the review. > Add support for Node Labels in opportunistic scheduling. > > > Key: YARN-8163 > URL: https://issues.apache.org/jira/browse/YARN-8163 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8163.002.patch, YARN-8163.003.patch, > YARN-8163.004.patch, YARN-8163.patch > > > Currently, the Opportunistic Scheduler doesn't honor node labels constraints > and schedule containers based on locality and load constraints. This Jira is > to add support in opportunistic scheduling to honor node labels in resource > requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8275) Create a JNI interface to interact with Windows
[ https://issues.apache.org/jira/browse/YARN-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474424#comment-16474424 ] Íñigo Goiri commented on YARN-8275: --- [~aw], thanks for the feedback, much appreciated. It looks like we can put all you proposed together into an umbrella for fixing the way Hadoop interacts with Windows. >From this thread, I see: * Move away from an external processese (winutils.exe) for native code: ** Replace by native Java APIs (e.g., symlinks) ** Replace by something like JNI or so * Fix the build system to fully leverage cmake instead of msbuild I would create an umbrella for this bigger task and make this JIRA just a subtask focusing on the YARN side (e.g., task). > Create a JNI interface to interact with Windows > --- > > Key: YARN-8275 > URL: https://issues.apache.org/jira/browse/YARN-8275 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: WinUtils-Functions.pdf, WinUtils.CSV > > > I did a quick investigation of the performance of WinUtils in YARN. In > average NM calls 4.76 times per second and 65.51 per container. > > | |Requests|Requests/sec|Requests/min|Requests/container| > |*Sum [WinUtils]*|*135354*|*4.761*|*286.160*|*65.51*| > |[WinUtils] Execute -help|4148|0.145|8.769|2.007| > |[WinUtils] Execute -ls|2842|0.0999|6.008|1.37| > |[WinUtils] Execute -systeminfo|9153|0.321|19.35|4.43| > |[WinUtils] Execute -symlink|115096|4.048|243.33|57.37| > |[WinUtils] Execute -task isAlive|4115|0.144|8.699|2.05| > Interval: 7 hours, 53 minutes and 48 seconds > Each execution of WinUtils does around *140 IO ops*, of which 130 are DDL ops. > This means *666.58* IO ops/second due to WinUtils. > We should start considering to remove WinUtils from Hadoop and creating a JNI > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8275) Create a JNI interface to interact with Windows
[ https://issues.apache.org/jira/browse/YARN-8275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472863#comment-16472863 ] Íñigo Goiri commented on YARN-8275: --- [~miklos.szeg...@cloudera.com], we would have to decide how to write the native service and that opens a big design space. Do you have any proposal for that? In any case, I would make this pluggable and then we can rely on winutils, a separate service or JNI. > Create a JNI interface to interact with Windows > --- > > Key: YARN-8275 > URL: https://issues.apache.org/jira/browse/YARN-8275 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: WinUtils-Functions.pdf, WinUtils.CSV > > > I did a quick investigation of the performance of WinUtils in YARN. In > average NM calls 4.76 times per second and 65.51 per container. > > | |Requests|Requests/sec|Requests/min|Requests/container| > |*Sum [WinUtils]*|*135354*|*4.761*|*286.160*|*65.51*| > |[WinUtils] Execute -help|4148|0.145|8.769|2.007| > |[WinUtils] Execute -ls|2842|0.0999|6.008|1.37| > |[WinUtils] Execute -systeminfo|9153|0.321|19.35|4.43| > |[WinUtils] Execute -symlink|115096|4.048|243.33|57.37| > |[WinUtils] Execute -task isAlive|4115|0.144|8.699|2.05| > Interval: 7 hours, 53 minutes and 48 seconds > Each execution of WinUtils does around *140 IO ops*, of which 130 are DDL ops. > This means *666.58* IO ops/second due to WinUtils. > We should start considering to remove WinUtils from Hadoop and creating a JNI > interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8276) [UI2] After version field became mandatory, form-based submission of new YARN service through UI2 doesn't work
Gergely Novák created YARN-8276:
---
Summary: [UI2] After version field became mandatory, form-based
submission of new YARN service through UI2 doesn't work
Key: YARN-8276
URL: https://issues.apache.org/jira/browse/YARN-8276
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-ui-v2
Reporter: Gergely Novák
Assignee: Gergely Novák
After version became mandatory in YARN service, one cannot create a new service
through UI, there is no way to specify the version field and the service fails
with the following message:
{code}
"Error: Adapter operation failed".
{code}
Checking through browser dev tools, the REST response is the following:
{code}
{"diagnostics":"Version of service sleeper-service is either empty or not
provided"}
{code}
Discovered by [~vinodkv].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8277) [UI2] Changing the user name for service in the new UI does not update the hadoop.auth cookie
Gergely Novák created YARN-8277:
---
Summary: [UI2] Changing the user name for service in the new UI
does not update the hadoop.auth cookie
Key: YARN-8277
URL: https://issues.apache.org/jira/browse/YARN-8277
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-ui-v2
Reporter: Gergely Novák
Steps to reproduce:
1) Click on "New Service" on the new YARN UI.
2) Provide an invalid, not existing user (e.g. "invalid").
3) Provide a service definition (e.g. sleeper) via the form or json.
4) Press deploy.
5) Response: Error: Adapter operation failed. With developer tools you can
check that {{/app/v1/services/?user.name=invalid}} gave a 403 status code with
this diagnostic message:
{code}
"Permission denied: user=invalid, access=EXECUTE, ...
{code}
6) Fix the user name, provide an existing user.
7) Press deploy.
Expected behaviour: the user should successfully deploy the service.
Current behaviour: The error response is the same.
Root cause: The query parameter of the new request is correct
({{?user.name=existing_user}}), however the hadoop.auth cookie is the old one:
{{"u=invalid=invalid=simple-dt=1526073119565...}}. Thus the request will
fail with the same "Permission denied" error.
The only workaround for the user to fix this is to clear the hadoop.auth cookie
from the browser.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8276) [UI2] After version field became mandatory, form-based submission of new YARN service through UI2 doesn't work
[
https://issues.apache.org/jira/browse/YARN-8276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gergely Novák updated YARN-8276:
Attachment: YARN-8276.001.patch
> [UI2] After version field became mandatory, form-based submission of new YARN
> service through UI2 doesn't work
> --
>
> Key: YARN-8276
> URL: https://issues.apache.org/jira/browse/YARN-8276
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Gergely Novák
>Assignee: Gergely Novák
>Priority: Major
> Attachments: YARN-8276.001.patch
>
>
> After version became mandatory in YARN service, one cannot create a new
> service through UI, there is no way to specify the version field and the
> service fails with the following message:
> {code}
> "Error: Adapter operation failed".
> {code}
> Checking through browser dev tools, the REST response is the following:
> {code}
> {"diagnostics":"Version of service sleeper-service is either empty or not
> provided"}
> {code}
> Discovered by [~vinodkv].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8280) [UI2] GPU is not present in the drop down box under 'Nodes Heatmap'
[ https://issues.apache.org/jira/browse/YARN-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472022#comment-16472022 ] Gergely Novák commented on YARN-8280: - Uploaded WiP patch, works locally but I'd like to test it on a cluster, too. > [UI2] GPU is not present in the drop down box under 'Nodes Heatmap' > --- > > Key: YARN-8280 > URL: https://issues.apache.org/jira/browse/YARN-8280 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Gergely Novák >Assignee: Gergely Novák >Priority: Major > Attachments: YARN-8280.001.patch > > > 1. Click on Nodes Heatmap Chart under Nodes Tab. > 2. No option to select GPU is available under the drop down menu. > Discovered by [~ssath...@hortonworks.com]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8280) [UI2] GPU is not present in the drop down box under 'Nodes Heatmap'
[ https://issues.apache.org/jira/browse/YARN-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated YARN-8280: Attachment: YARN-8280.001.patch > [UI2] GPU is not present in the drop down box under 'Nodes Heatmap' > --- > > Key: YARN-8280 > URL: https://issues.apache.org/jira/browse/YARN-8280 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Gergely Novák >Assignee: Gergely Novák >Priority: Major > Attachments: YARN-8280.001.patch > > > 1. Click on Nodes Heatmap Chart under Nodes Tab. > 2. No option to select GPU is available under the drop down menu. > Discovered by [~ssath...@hortonworks.com]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
