[jira] [Created] (YARN-6652) Merge flow info and flow runs
Haibo Chen created YARN-6652: Summary: Merge flow info and flow runs Key: YARN-6652 URL: https://issues.apache.org/jira/browse/YARN-6652 Project: Hadoop YARN Issue Type: Improvement Components: yarn-ui-v2 Affects Versions: 3.0.0-alpha2 Reporter: Haibo Chen If a user clicks on a flow from the flow activity page, Flow Run and Flow Info are shown separately. Usually, users want to go to individual flow runs. With the current work flow, the user will need to click on Flow Run because Flow Info is selected by default. Given that Flow Info does not have much information, It'd be a nice improvement if we can show flow info and flow run together, that is, one section at the top containing flow info, another section at the bottom containing the flow runs -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6555) Enable flow context read (& corresponding write) for recovering application with NM restart
[
https://issues.apache.org/jira/browse/YARN-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025390#comment-16025390
]
Haibo Chen commented on YARN-6555:
--
Thanks [~rohithsharma] for the explanation, and updating the patch! +1 on the
latest patch. [~vrushalic] do you have any other comments?
> Enable flow context read (& corresponding write) for recovering application
> with NM restart
>
>
> Key: YARN-6555
> URL: https://issues.apache.org/jira/browse/YARN-6555
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha3
>Reporter: Vrushali C
>Assignee: Rohith Sharma K S
> Labels: yarn-5355-merge-blocker
> Attachments: YARN-6555.001.patch, YARN-6555.002.patch,
> YARN-6555.003.patch
>
>
> If timeline service v2 is enabled and NM is restarted with recovery enabled,
> then NM fails to start and throws an error as "flow context can't be null".
> This is happening because the flow context did not exist before but now that
> timeline service v2 is enabled, ApplicationImpl expects it to exist.
> This would also happen even if flow context existed before but since we are
> not persisting it / reading it during
> ContainerManagerImpl#recoverApplication, it does not get passed in to
> ApplicationImpl.
> full stack trace
> {code}
> 2017-05-03 21:51:52,178 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> java.lang.IllegalArgumentException: flow context cannot be null
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:90)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverApplication(ContainerManagerImpl.java:318)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:280)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:267)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:276)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:588)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:649)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6555) Store application flow context in NM state store for work-preserving restart
[
https://issues.apache.org/jira/browse/YARN-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-6555:
-
Fix Version/s: 3.0.0-alpha3
> Store application flow context in NM state store for work-preserving restart
>
>
> Key: YARN-6555
> URL: https://issues.apache.org/jira/browse/YARN-6555
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha3
>Reporter: Vrushali C
>Assignee: Rohith Sharma K S
> Labels: yarn-5355-merge-blocker
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6555.001.patch, YARN-6555.002.patch,
> YARN-6555.003.patch
>
>
> If timeline service v2 is enabled and NM is restarted with recovery enabled,
> then NM fails to start and throws an error as "flow context can't be null".
> This is happening because the flow context did not exist before but now that
> timeline service v2 is enabled, ApplicationImpl expects it to exist.
> This would also happen even if flow context existed before but since we are
> not persisting it / reading it during
> ContainerManagerImpl#recoverApplication, it does not get passed in to
> ApplicationImpl.
> full stack trace
> {code}
> 2017-05-03 21:51:52,178 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> java.lang.IllegalArgumentException: flow context cannot be null
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:90)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverApplication(ContainerManagerImpl.java:318)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:280)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:267)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:276)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:588)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:649)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6555) Store application flow context in NM state store for work-preserving restart
[
https://issues.apache.org/jira/browse/YARN-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-6555:
-
Fix Version/s: YARN-5355-branch-2
YARN-5355
> Store application flow context in NM state store for work-preserving restart
>
>
> Key: YARN-6555
> URL: https://issues.apache.org/jira/browse/YARN-6555
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha3
>Reporter: Vrushali C
>Assignee: Rohith Sharma K S
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha3
>
> Attachments: YARN-6555.001.patch, YARN-6555.002.patch,
> YARN-6555.003.patch
>
>
> If timeline service v2 is enabled and NM is restarted with recovery enabled,
> then NM fails to start and throws an error as "flow context can't be null".
> This is happening because the flow context did not exist before but now that
> timeline service v2 is enabled, ApplicationImpl expects it to exist.
> This would also happen even if flow context existed before but since we are
> not persisting it / reading it during
> ContainerManagerImpl#recoverApplication, it does not get passed in to
> ApplicationImpl.
> full stack trace
> {code}
> 2017-05-03 21:51:52,178 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> java.lang.IllegalArgumentException: flow context cannot be null
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:90)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverApplication(ContainerManagerImpl.java:318)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:280)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:267)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:276)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:588)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:649)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6555) Store application flow context in NM state store for work-preserving restart
[
https://issues.apache.org/jira/browse/YARN-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-6555:
-
Summary: Store application flow context in NM state store for
work-preserving restart (was: Enable flow context read (& corresponding write)
for recovering application with NM restart )
> Store application flow context in NM state store for work-preserving restart
>
>
> Key: YARN-6555
> URL: https://issues.apache.org/jira/browse/YARN-6555
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha3
>Reporter: Vrushali C
>Assignee: Rohith Sharma K S
> Labels: yarn-5355-merge-blocker
> Attachments: YARN-6555.001.patch, YARN-6555.002.patch,
> YARN-6555.003.patch
>
>
> If timeline service v2 is enabled and NM is restarted with recovery enabled,
> then NM fails to start and throws an error as "flow context can't be null".
> This is happening because the flow context did not exist before but now that
> timeline service v2 is enabled, ApplicationImpl expects it to exist.
> This would also happen even if flow context existed before but since we are
> not persisting it / reading it during
> ContainerManagerImpl#recoverApplication, it does not get passed in to
> ApplicationImpl.
> full stack trace
> {code}
> 2017-05-03 21:51:52,178 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> java.lang.IllegalArgumentException: flow context cannot be null
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:90)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverApplication(ContainerManagerImpl.java:318)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:280)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:267)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:276)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:588)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:649)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6555) Store application flow context in NM state store for work-preserving restart
[
https://issues.apache.org/jira/browse/YARN-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025843#comment-16025843
]
Haibo Chen commented on YARN-6555:
--
Yes, I already committed this into trunk
> Store application flow context in NM state store for work-preserving restart
>
>
> Key: YARN-6555
> URL: https://issues.apache.org/jira/browse/YARN-6555
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha3
>Reporter: Vrushali C
>Assignee: Rohith Sharma K S
> Labels: yarn-5355-merge-blocker
> Fix For: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha3
>
> Attachments: YARN-6555.001.patch, YARN-6555.002.patch,
> YARN-6555.003.patch
>
>
> If timeline service v2 is enabled and NM is restarted with recovery enabled,
> then NM fails to start and throws an error as "flow context can't be null".
> This is happening because the flow context did not exist before but now that
> timeline service v2 is enabled, ApplicationImpl expects it to exist.
> This would also happen even if flow context existed before but since we are
> not persisting it / reading it during
> ContainerManagerImpl#recoverApplication, it does not get passed in to
> ApplicationImpl.
> full stack trace
> {code}
> 2017-05-03 21:51:52,178 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> java.lang.IllegalArgumentException: flow context cannot be null
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:90)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverApplication(ContainerManagerImpl.java:318)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:280)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:267)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:276)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:588)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:649)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5705) [YARN-3368] Add support for Timeline V2 to new web UI
[ https://issues.apache.org/jira/browse/YARN-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016215#comment-16016215 ] Haibo Chen commented on YARN-5705: -- Thanks [~akhilpb] for the update! I played with the latests patch with an ATSv2-enabled YARN cluster. The UI looks good. One issue I noticed is that the cpu vcores & memory used are always zero for flow run. > [YARN-3368] Add support for Timeline V2 to new web UI > - > > Key: YARN-5705 > URL: https://issues.apache.org/jira/browse/YARN-5705 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sunil G >Assignee: Akhil PB > Labels: oct16-hard > Attachments: Screenshots.zip, YARN-5705.001.patch, > YARN-5705.002.patch, YARN-5705.003.patch, YARN-5705.004.patch, > YARN-5705.005.patch, YARN-5705.006.patch, YARN-5705.007.patch, > YARN-5705.008.patch, YARN-5705.009.patch, YARN-5705.010.patch, > YARN-5705.011.patch, YARN-5705.012.patch, YARN-5705.013.patch, > YARN-5705.014.patch, YARN-5705.015.patch, YARN-5705.016.patch, > YARN-5705-YARN-3368.001.patch, YARN-5705-YARN-3368.002.patch, > YARN-5705-YARN-3368.003.patch, YARN-5705-YARN-3368.004.patch > > > Integrate timeline v2 to YARN-3368. This is a clone JIRA for YARN-4097 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6670) Add over-allocation thresholds to allow per-node oversubscription
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6670: - Summary: Add over-allocation thresholds to allow per-node oversubscription (was: Add over-allocation threshold to allow per-node oversubscription) > Add over-allocation thresholds to allow per-node oversubscription > - > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6671) Add container type awareness in ResourceHandlers.
Haibo Chen created YARN-6671: Summary: Add container type awareness in ResourceHandlers. Key: YARN-6671 URL: https://issues.apache.org/jira/browse/YARN-6671 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 3.0.0-alpha3 Reporter: Haibo Chen Assignee: Miklos Szegedi When using LCE, different cgroup settings for opportunistic and guaranteed containers can be used to ensure isolation between them. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6674) Add memory cgroup settings for opportunistic containers
Haibo Chen created YARN-6674: Summary: Add memory cgroup settings for opportunistic containers Key: YARN-6674 URL: https://issues.apache.org/jira/browse/YARN-6674 Project: Hadoop YARN Issue Type: Task Components: nodemanager Affects Versions: 3.0.0-alpha3 Reporter: Haibo Chen Assignee: Miklos Szegedi -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6656) The column family indicated in AppToFlowTable class javadoc is wrong
[ https://issues.apache.org/jira/browse/YARN-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen resolved YARN-6656. -- Resolution: Duplicate > The column family indicated in AppToFlowTable class javadoc is wrong > > > Key: YARN-6656 > URL: https://issues.apache.org/jira/browse/YARN-6656 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Trivial > > AppToFlowTable javadoc says the column family is info, whereas in reality it > is mapping. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6668) Use cgroup to get container resource utilization
Haibo Chen created YARN-6668: Summary: Use cgroup to get container resource utilization Key: YARN-6668 URL: https://issues.apache.org/jira/browse/YARN-6668 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 3.0.0-alpha3 Reporter: Haibo Chen Assignee: Haibo Chen Container Monitor relies on proc file system to get container resource utilization, which is not as efficient as reading cgroup accounting. We should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6671) Add container type awareness in ResourceHandlers.
[ https://issues.apache.org/jira/browse/YARN-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6671: - Issue Type: Sub-task (was: Improvement) Parent: YARN-1011 > Add container type awareness in ResourceHandlers. > - > > Key: YARN-6671 > URL: https://issues.apache.org/jira/browse/YARN-6671 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > > When using LCE, different cgroup settings for opportunistic and guaranteed > containers can be used to ensure isolation between them. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6670) Add over-allocation threshold to allow per-node oversubscription
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6670: - Summary: Add over-allocation threshold to allow per-node oversubscription (was: Add a new configuration, over-allocation threshold, to allow per-node oversubscription) > Add over-allocation threshold to allow per-node oversubscription > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6672) Add NM preemption of opportunistic containers
[ https://issues.apache.org/jira/browse/YARN-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6672: - Issue Type: Sub-task (was: Task) Parent: YARN-1011 > Add NM preemption of opportunistic containers > - > > Key: YARN-6672 > URL: https://issues.apache.org/jira/browse/YARN-6672 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6672) Add NM preemption of opportunistic containers
Haibo Chen created YARN-6672: Summary: Add NM preemption of opportunistic containers Key: YARN-6672 URL: https://issues.apache.org/jira/browse/YARN-6672 Project: Hadoop YARN Issue Type: Task Affects Versions: 3.0.0-alpha3 Reporter: Haibo Chen Assignee: Haibo Chen -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen reassigned YARN-6668: Assignee: Miklos Szegedi (was: Haibo Chen) > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6670) Add a new configuration, over-allocation threshold, to allow per-node oversubscription
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6670: - Issue Type: Sub-task (was: Improvement) Parent: YARN-1011 > Add a new configuration, over-allocation threshold, to allow per-node > oversubscription > -- > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6668: - Issue Type: Sub-task (was: Improvement) Parent: YARN-1011 > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6674) Add memory cgroup settings for opportunistic containers
[ https://issues.apache.org/jira/browse/YARN-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6674: - Issue Type: Sub-task (was: Task) Parent: YARN-1011 > Add memory cgroup settings for opportunistic containers > --- > > Key: YARN-6674 > URL: https://issues.apache.org/jira/browse/YARN-6674 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6673) Add cpu cgroup configurations for opportunistic containers
Haibo Chen created YARN-6673: Summary: Add cpu cgroup configurations for opportunistic containers Key: YARN-6673 URL: https://issues.apache.org/jira/browse/YARN-6673 Project: Hadoop YARN Issue Type: Task Reporter: Haibo Chen Assignee: Miklos Szegedi -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6673) Add cpu cgroup configurations for opportunistic containers
[ https://issues.apache.org/jira/browse/YARN-6673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6673: - Issue Type: Sub-task (was: Task) Parent: YARN-1011 > Add cpu cgroup configurations for opportunistic containers > -- > > Key: YARN-6673 > URL: https://issues.apache.org/jira/browse/YARN-6673 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Haibo Chen >Assignee: Miklos Szegedi > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6672) Add NM preemption of opportunistic containers when utilization goes high
[ https://issues.apache.org/jira/browse/YARN-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6672: - Summary: Add NM preemption of opportunistic containers when utilization goes high (was: Add NM preemption of opportunistic containers) > Add NM preemption of opportunistic containers when utilization goes high > > > Key: YARN-6672 > URL: https://issues.apache.org/jira/browse/YARN-6672 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6657) The column family indicated in AppToFlowTable class javadoc is wrong
[ https://issues.apache.org/jira/browse/YARN-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028724#comment-16028724 ] Haibo Chen commented on YARN-6657: -- Thanks for pointing it out, [~varun_saxena]. Uploading the correct patch > The column family indicated in AppToFlowTable class javadoc is wrong > > > Key: YARN-6657 > URL: https://issues.apache.org/jira/browse/YARN-6657 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Trivial > Attachments: YARN-6657.00.patch, YARN-6658.00.patch > > > AppToFlowTable javadoc says the column family is info, whereas in reality it > is mapping. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6657) The column family indicated in AppToFlowTable class javadoc is wrong
[ https://issues.apache.org/jira/browse/YARN-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6657: - Attachment: YARN-6657.00.patch > The column family indicated in AppToFlowTable class javadoc is wrong > > > Key: YARN-6657 > URL: https://issues.apache.org/jira/browse/YARN-6657 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Trivial > Attachments: YARN-6657.00.patch, YARN-6658.00.patch > > > AppToFlowTable javadoc says the column family is info, whereas in reality it > is mapping. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6675) Add NM support to launch opportunistic containers based on oversubscription
Haibo Chen created YARN-6675: Summary: Add NM support to launch opportunistic containers based on oversubscription Key: YARN-6675 URL: https://issues.apache.org/jira/browse/YARN-6675 Project: Hadoop YARN Issue Type: Task Affects Versions: 3.0.0-alpha3 Reporter: Haibo Chen Assignee: Haibo Chen -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6675) Add NM support to launch opportunistic containers based on oversubscription
[ https://issues.apache.org/jira/browse/YARN-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6675: - Issue Type: Sub-task (was: Task) Parent: YARN-1011 > Add NM support to launch opportunistic containers based on oversubscription > --- > > Key: YARN-6675 > URL: https://issues.apache.org/jira/browse/YARN-6675 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6477) Dispatcher no longer needs the raw types suppression
[
https://issues.apache.org/jira/browse/YARN-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030393#comment-16030393
]
Haibo Chen commented on YARN-6477:
--
Thanks [~maya] for your contribution! I have committed it to trunk.
> Dispatcher no longer needs the raw types suppression
>
>
> Key: YARN-6477
> URL: https://issues.apache.org/jira/browse/YARN-6477
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Maya Wexler
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6477.001.patch
>
>
> Post YARN-4457, the {{@SuppressWarnings("rawtypes")}} is no longer needed in
> the Dispatcher class.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6477) Dispatcher no longer needs the raw types suppression
[
https://issues.apache.org/jira/browse/YARN-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030385#comment-16030385
]
Haibo Chen commented on YARN-6477:
--
+1. Checking this in.
> Dispatcher no longer needs the raw types suppression
>
>
> Key: YARN-6477
> URL: https://issues.apache.org/jira/browse/YARN-6477
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Maya Wexler
>Priority: Minor
> Labels: newbie
> Attachments: YARN-6477.001.patch
>
>
> Post YARN-4457, the {{@SuppressWarnings("rawtypes")}} is no longer needed in
> the Dispatcher class.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6658) Remove columnFor() methods of Columns in HBaseTimeline backend
[ https://issues.apache.org/jira/browse/YARN-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029782#comment-16029782 ] Haibo Chen commented on YARN-6658: -- This is more of a minor cleanup. Given that this will eventually go into trunk with the merge, we can just get it in YARN-5355. > Remove columnFor() methods of Columns in HBaseTimeline backend > --- > > Key: YARN-6658 > URL: https://issues.apache.org/jira/browse/YARN-6658 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6658.00.patch > > > The columnFor() methods are all dead. We can remove them. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5094) some YARN container events have timestamp of -1
[
https://issues.apache.org/jira/browse/YARN-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-5094:
-
Attachment: YARN-5094.00.patch
> some YARN container events have timestamp of -1
> ---
>
> Key: YARN-5094
> URL: https://issues.apache.org/jira/browse/YARN-5094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
> Labels: YARN-5355
> Attachments: YARN-5094.00.patch, YARN-5094-YARN-2928.001.patch
>
>
> Some events in the YARN container entities have timestamp of -1. The
> RM-generated container events have proper timestamps. It appears that it's
> the NM-generated events that have -1: YARN_CONTAINER_CREATED,
> YARN_CONTAINER_FINISHED, YARN_NM_CONTAINER_LOCALIZATION_FINISHED,
> YARN_NM_CONTAINER_LOCALIZATION_STARTED.
> In the YARN container page,
> {noformat}
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: -1,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: -1,
> info: { }
> }
> {noformat}
> I think the data itself is OK, but the values are not being populated in the
> REST output?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033932#comment-16033932 ] Haibo Chen commented on YARN-6670: -- Findbug warnings are unrelated. Will update the patch to fix the checkstyle issues. > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5094) some YARN container events have timestamp of -1
[
https://issues.apache.org/jira/browse/YARN-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033935#comment-16033935
]
Haibo Chen commented on YARN-5094:
--
Thanks [~gtCarrera9] for your comments. Yes, I have limited the change to only
events that we are interested in exposing to ATSv2 for the exact same reason
that you pointed.
> some YARN container events have timestamp of -1
> ---
>
> Key: YARN-5094
> URL: https://issues.apache.org/jira/browse/YARN-5094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
> Labels: YARN-5355
> Attachments: YARN-5094.00.patch, YARN-5094-YARN-2928.001.patch
>
>
> Some events in the YARN container entities have timestamp of -1. The
> RM-generated container events have proper timestamps. It appears that it's
> the NM-generated events that have -1: YARN_CONTAINER_CREATED,
> YARN_CONTAINER_FINISHED, YARN_NM_CONTAINER_LOCALIZATION_FINISHED,
> YARN_NM_CONTAINER_LOCALIZATION_STARTED.
> In the YARN container page,
> {noformat}
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: -1,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: -1,
> info: { }
> }
> {noformat}
> I think the data itself is OK, but the values are not being populated in the
> REST output?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6316) Provide help information and documentation for TimelineSchemaCreator
[ https://issues.apache.org/jira/browse/YARN-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033937#comment-16033937 ] Haibo Chen commented on YARN-6316: -- Agreed. Thanks [~vrushalic]! > Provide help information and documentation for TimelineSchemaCreator > > > Key: YARN-6316 > URL: https://issues.apache.org/jira/browse/YARN-6316 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Haibo Chen > Attachments: YARN-6316.00.patch, YARN-6316.prelim.patch > > > Right now there is no help information for timeline schema creator. We may > probably want to provide an option to print help. Also, ideally, if users > passed in no argument, we may want to print out help, instead of directly > create the tables. This will simplify cluster operations and timeline v2 > deployments. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5094) some YARN container events have timestamp of -1
[
https://issues.apache.org/jira/browse/YARN-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033878#comment-16033878
]
Haibo Chen commented on YARN-5094:
--
Sorry, I just realized that AbstractEvent is used in everywhere. To be
conservative, I think we can set the timestamp for events only when we actually
need the timestamps (e.g. events exposed in ATSv2)
Thus, I am uploading a patch here to only change NM
container/localization/application event timestamps.
> some YARN container events have timestamp of -1
> ---
>
> Key: YARN-5094
> URL: https://issues.apache.org/jira/browse/YARN-5094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
> Labels: YARN-5355
> Attachments: YARN-5094-YARN-2928.001.patch
>
>
> Some events in the YARN container entities have timestamp of -1. The
> RM-generated container events have proper timestamps. It appears that it's
> the NM-generated events that have -1: YARN_CONTAINER_CREATED,
> YARN_CONTAINER_FINISHED, YARN_NM_CONTAINER_LOCALIZATION_FINISHED,
> YARN_NM_CONTAINER_LOCALIZATION_STARTED.
> In the YARN container page,
> {noformat}
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: -1,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: -1,
> info: { }
> }
> {noformat}
> I think the data itself is OK, but the values are not being populated in the
> REST output?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5094) some YARN container events have timestamp of -1 in REST output
[
https://issues.apache.org/jira/browse/YARN-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen reassigned YARN-5094:
Assignee: Haibo Chen (was: Li Lu)
> some YARN container events have timestamp of -1 in REST output
> --
>
> Key: YARN-5094
> URL: https://issues.apache.org/jira/browse/YARN-5094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
> Labels: YARN-5355
> Attachments: YARN-5094-YARN-2928.001.patch
>
>
> Some events in the YARN container entities have timestamp of -1. The
> RM-generated container events have proper timestamps. It appears that it's
> the NM-generated events that have -1: YARN_CONTAINER_CREATED,
> YARN_CONTAINER_FINISHED, YARN_NM_CONTAINER_LOCALIZATION_FINISHED,
> YARN_NM_CONTAINER_LOCALIZATION_STARTED.
> In the YARN container page,
> {noformat}
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: -1,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: -1,
> info: { }
> }
> {noformat}
> I think the data itself is OK, but the values are not being populated in the
> REST output?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6199) Support for listing flows with filter userid
[ https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen reassigned YARN-6199: Assignee: Haibo Chen > Support for listing flows with filter userid > > > Key: YARN-6199 > URL: https://issues.apache.org/jira/browse/YARN-6199 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > > Currently */flows* API retrieves flow entities for all the users by default. > It is required to provide filter user i.e */flows?user=rohith* . This is > critical filter in secured environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6604) Allow metric TTL for Application table to be specified through cmd
[ https://issues.apache.org/jira/browse/YARN-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6604: - Attachment: YARN-6604.00.patch > Allow metric TTL for Application table to be specified through cmd > -- > > Key: YARN-6604 > URL: https://issues.apache.org/jira/browse/YARN-6604 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.0.0-alpha2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6604.00.patch > > > We should allow metrics TTL in application table to be specified in schema > cmd, as we do with metrics TTL in entity table -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6670: - Attachment: YARN-6670-YARN-1011.00.patch > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6670) Add separate oversubscription thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6670: - Summary: Add separate oversubscription thresholds for cpu and memory (was: Add separate configurations for cpu and memory oversubscription) > Add separate oversubscription thresholds for cpu and memory > --- > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6670: - Summary: Add separate NM overallocation thresholds for cpu and memory (was: Add separate oversubscription thresholds for cpu and memory) > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5094) some YARN container events have timestamp of -1
[
https://issues.apache.org/jira/browse/YARN-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-5094:
-
Summary: some YARN container events have timestamp of -1 (was: some YARN
container events have timestamp of -1 in REST output)
> some YARN container events have timestamp of -1
> ---
>
> Key: YARN-5094
> URL: https://issues.apache.org/jira/browse/YARN-5094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
> Labels: YARN-5355
> Attachments: YARN-5094-YARN-2928.001.patch
>
>
> Some events in the YARN container entities have timestamp of -1. The
> RM-generated container events have proper timestamps. It appears that it's
> the NM-generated events that have -1: YARN_CONTAINER_CREATED,
> YARN_CONTAINER_FINISHED, YARN_NM_CONTAINER_LOCALIZATION_FINISHED,
> YARN_NM_CONTAINER_LOCALIZATION_STARTED.
> In the YARN container page,
> {noformat}
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: -1,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: -1,
> info: { }
> }
> {noformat}
> I think the data itself is OK, but the values are not being populated in the
> REST output?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5094) some YARN container events have timestamp of -1
[
https://issues.apache.org/jira/browse/YARN-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035116#comment-16035116
]
Haibo Chen commented on YARN-5094:
--
Will update the patch to address the unit test failures.
> some YARN container events have timestamp of -1
> ---
>
> Key: YARN-5094
> URL: https://issues.apache.org/jira/browse/YARN-5094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
> Labels: YARN-5355
> Attachments: YARN-5094.00.patch, YARN-5094-YARN-2928.001.patch
>
>
> Some events in the YARN container entities have timestamp of -1. The
> RM-generated container events have proper timestamps. It appears that it's
> the NM-generated events that have -1: YARN_CONTAINER_CREATED,
> YARN_CONTAINER_FINISHED, YARN_NM_CONTAINER_LOCALIZATION_FINISHED,
> YARN_NM_CONTAINER_LOCALIZATION_STARTED.
> In the YARN container page,
> {noformat}
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: -1,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: -1,
> info: { }
> }
> {noformat}
> I think the data itself is OK, but the values are not being populated in the
> REST output?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5094) some YARN container events have timestamp of -1
[
https://issues.apache.org/jira/browse/YARN-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-5094:
-
Attachment: YARN-5094.02.patch
Patch updated to ignore ApplicationLocalizationEvent timestamp when matching
arguments
> some YARN container events have timestamp of -1
> ---
>
> Key: YARN-5094
> URL: https://issues.apache.org/jira/browse/YARN-5094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
> Labels: YARN-5355
> Attachments: YARN-5094.00.patch, YARN-5094.02.patch,
> YARN-5094-YARN-2928.001.patch
>
>
> Some events in the YARN container entities have timestamp of -1. The
> RM-generated container events have proper timestamps. It appears that it's
> the NM-generated events that have -1: YARN_CONTAINER_CREATED,
> YARN_CONTAINER_FINISHED, YARN_NM_CONTAINER_LOCALIZATION_FINISHED,
> YARN_NM_CONTAINER_LOCALIZATION_STARTED.
> In the YARN container page,
> {noformat}
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: -1,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: -1,
> info: { }
> }
> {noformat}
> I think the data itself is OK, but the values are not being populated in the
> REST output?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5169) most YARN events have timestamp of -1
[
https://issues.apache.org/jira/browse/YARN-5169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen reassigned YARN-5169:
Assignee: (was: Haibo Chen)
> most YARN events have timestamp of -1
> -
>
> Key: YARN-5169
> URL: https://issues.apache.org/jira/browse/YARN-5169
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 2.7.2
>Reporter: Sangjin Lee
>
> Most of the YARN events (subclasses of {{AbstractEvent}}) have timestamp of
> -1. {{AbstractEvent}} have two constructors, one that initializes the
> timestamp to -1 and the other to the caller-provided value. But most events
> use the former (thus timestamp of -1).
> Some of the more common events, including {{ApplicationEvent}},
> {{ContainerEvent}}, {{JobEvent}}, etc. do not set the timestamp.
> The rationale for this behavior seems to be mentioned in {{AbstractEvent}}:
> {code}
> // use this if you DON'T care about the timestamp
> public AbstractEvent(TYPE type) {
> this.type = type;
> // We're not generating a real timestamp here. It's too expensive.
> timestamp = -1L;
> }
> {code}
> This absence of the timestamp isn't really visible in many cases and
> therefore may have gone unnoticed, but the timeline service exposes this
> problem very visibly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6047) Documentation updates for TimelineService v2
[ https://issues.apache.org/jira/browse/YARN-6047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035160#comment-16035160 ] Haibo Chen commented on YARN-6047: -- Linked YARN-6382 to document the known writer fault-tolerance issue. > Documentation updates for TimelineService v2 > > > Key: YARN-6047 > URL: https://issues.apache.org/jira/browse/YARN-6047 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation, timelineserver >Reporter: Varun Saxena >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6047-YARN-5355.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6199) Support for listing flows with filter userid
[ https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035588#comment-16035588 ] Haibo Chen commented on YARN-6199: -- The current /flows query is implemented by setting the scan range. We could add a RowFilter with a regex matching the user id to the scan. > Support for listing flows with filter userid > > > Key: YARN-6199 > URL: https://issues.apache.org/jira/browse/YARN-6199 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > > Currently */flows* API retrieves flow entities for all the users by default. > It is required to provide filter user i.e */flows?user=rohith* . This is > critical filter in secured environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5094) some YARN container events have timestamp of -1
[
https://issues.apache.org/jira/browse/YARN-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035301#comment-16035301
]
Haibo Chen commented on YARN-5094:
--
Will look into it and follow up with a new patch
> some YARN container events have timestamp of -1
> ---
>
> Key: YARN-5094
> URL: https://issues.apache.org/jira/browse/YARN-5094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
> Labels: YARN-5355
> Attachments: YARN-5094.00.patch, YARN-5094.02.patch,
> YARN-5094-YARN-2928.001.patch
>
>
> Some events in the YARN container entities have timestamp of -1. The
> RM-generated container events have proper timestamps. It appears that it's
> the NM-generated events that have -1: YARN_CONTAINER_CREATED,
> YARN_CONTAINER_FINISHED, YARN_NM_CONTAINER_LOCALIZATION_FINISHED,
> YARN_NM_CONTAINER_LOCALIZATION_STARTED.
> In the YARN container page,
> {noformat}
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: -1,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: -1,
> info: { }
> }
> {noformat}
> I think the data itself is OK, but the values are not being populated in the
> REST output?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5094) some YARN container events have timestamp of -1
[
https://issues.apache.org/jira/browse/YARN-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035282#comment-16035282
]
Haibo Chen commented on YARN-5094:
--
Findbug warnings are unrelated.
> some YARN container events have timestamp of -1
> ---
>
> Key: YARN-5094
> URL: https://issues.apache.org/jira/browse/YARN-5094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
> Labels: YARN-5355
> Attachments: YARN-5094.00.patch, YARN-5094.02.patch,
> YARN-5094-YARN-2928.001.patch
>
>
> Some events in the YARN container entities have timestamp of -1. The
> RM-generated container events have proper timestamps. It appears that it's
> the NM-generated events that have -1: YARN_CONTAINER_CREATED,
> YARN_CONTAINER_FINISHED, YARN_NM_CONTAINER_LOCALIZATION_FINISHED,
> YARN_NM_CONTAINER_LOCALIZATION_STARTED.
> In the YARN container page,
> {noformat}
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: -1,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: -1,
> info: { }
> }
> {noformat}
> I think the data itself is OK, but the values are not being populated in the
> REST output?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6199) Support for listing flows with filter userid
[
https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041192#comment-16041192
]
Haibo Chen edited comment on YARN-6199 at 6/7/17 4:46 PM:
--
The new patch uses SubstringComparator to match \${userId}! in the row key.
To keep the key encoding logic at the same place
(FlowActivityRowKey.FlowActivityRowKeyConverter), a new method is added to
FlowActivityRowKeyConverter to return the encoded component that is composed of
${userId} and a trailing separator
was (Author: haibochen):
The new patch uses SubstringComparator to match ${userId}! in the row key.
To keep the key encoding logic at the same place
(FlowActivityRowKey.FlowActivityRowKeyConverter), a new method is added to
FlowActivityRowKeyConverter to return the encoded ${userId}\!
> Support for listing flows with filter userid
>
>
> Key: YARN-6199
> URL: https://issues.apache.org/jira/browse/YARN-6199
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Haibo Chen
> Attachments: YARN-6199.00.patch, YARN-6199.01.patch
>
>
> Currently */flows* API retrieves flow entities for all the users by default.
> It is required to provide filter user i.e */flows?user=rohith* . This is
> critical filter in secured environment.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6199) Support for listing flows with filter userid
[
https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-6199:
-
Attachment: YARN-6199.01.patch
The new patch uses SubstringComparator to match {userId}! in the row key.
To keep the key encoding logic at the same place
(FlowActivityRowKey.FlowActivityRowKeyConverter), a new method is added to
FlowActivityRowKeyConverter to return the encoded {userId}!
> Support for listing flows with filter userid
>
>
> Key: YARN-6199
> URL: https://issues.apache.org/jira/browse/YARN-6199
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Haibo Chen
> Attachments: YARN-6199.00.patch, YARN-6199.01.patch
>
>
> Currently */flows* API retrieves flow entities for all the users by default.
> It is required to provide filter user i.e */flows?user=rohith* . This is
> critical filter in secured environment.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6199) Support for listing flows with filter userid
[
https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041192#comment-16041192
]
Haibo Chen edited comment on YARN-6199 at 6/7/17 4:45 PM:
--
The new patch uses SubstringComparator to match ${userId}! in the row key.
To keep the key encoding logic at the same place
(FlowActivityRowKey.FlowActivityRowKeyConverter), a new method is added to
FlowActivityRowKeyConverter to return the encoded ${userId}\!
was (Author: haibochen):
The new patch uses SubstringComparator to match {userId}! in the row key.
To keep the key encoding logic at the same place
(FlowActivityRowKey.FlowActivityRowKeyConverter), a new method is added to
FlowActivityRowKeyConverter to return the encoded {userId}!
> Support for listing flows with filter userid
>
>
> Key: YARN-6199
> URL: https://issues.apache.org/jira/browse/YARN-6199
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Haibo Chen
> Attachments: YARN-6199.00.patch, YARN-6199.01.patch
>
>
> Currently */flows* API retrieves flow entities for all the users by default.
> It is required to provide filter user i.e */flows?user=rohith* . This is
> critical filter in secured environment.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6199) Support for listing flows with filter userid
[ https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041109#comment-16041109 ] Haibo Chen commented on YARN-6199: -- bq. when startdate - specified, how long table scans happens? Does it happens full table scan? The date range get translated into the start/end row of the scan, so date range dictates how big the scan it. User id is just an addon filter that applies to the result of the scan determined by daterange if I understand correctly. So yes, it can be a full table scan if only a start date that way in the past is specified. Following the discussion above, I'll leave out daterange-related issues (YARN-4504, YARN-6179) and only address user id in this jira. > Support for listing flows with filter userid > > > Key: YARN-6199 > URL: https://issues.apache.org/jira/browse/YARN-6199 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Attachments: YARN-6199.00.patch > > > Currently */flows* API retrieves flow entities for all the users by default. > It is required to provide filter user i.e */flows?user=rohith* . This is > critical filter in secured environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6604) Allow metric TTL for Application table to be specified through cmd
[ https://issues.apache.org/jira/browse/YARN-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041210#comment-16041210 ] Haibo Chen commented on YARN-6604: -- Thanks [~varun_saxena] for your reviews! > Allow metric TTL for Application table to be specified through cmd > -- > > Key: YARN-6604 > URL: https://issues.apache.org/jira/browse/YARN-6604 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.0.0-alpha2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4 > > Attachments: YARN-6604.00.patch > > > We should allow metrics TTL in application table to be specified in schema > cmd, as we do with metrics TTL in entity table -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6199) Support for listing flows with filter userid
[
https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041192#comment-16041192
]
Haibo Chen edited comment on YARN-6199 at 6/7/17 4:46 PM:
--
The new patch uses SubstringComparator to match ${userId}! in the row key.
To keep the key encoding logic at the same place
(FlowActivityRowKey.FlowActivityRowKeyConverter), a new method is added to
FlowActivityRowKeyConverter to return the encoded component that is composed of
${userId} and a trailing separator
was (Author: haibochen):
The new patch uses SubstringComparator to match \${userId}! in the row key.
To keep the key encoding logic at the same place
(FlowActivityRowKey.FlowActivityRowKeyConverter), a new method is added to
FlowActivityRowKeyConverter to return the encoded component that is composed of
${userId} and a trailing separator
> Support for listing flows with filter userid
>
>
> Key: YARN-6199
> URL: https://issues.apache.org/jira/browse/YARN-6199
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Haibo Chen
> Attachments: YARN-6199.00.patch, YARN-6199.01.patch
>
>
> Currently */flows* API retrieves flow entities for all the users by default.
> It is required to provide filter user i.e */flows?user=rohith* . This is
> critical filter in secured environment.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6199) Support for listing flows with filter userid
[ https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041194#comment-16041194 ] Haibo Chen commented on YARN-6199: -- Will follow up with a patch for YARN-5355 > Support for listing flows with filter userid > > > Key: YARN-6199 > URL: https://issues.apache.org/jira/browse/YARN-6199 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Attachments: YARN-6199.00.patch, YARN-6199.01.patch > > > Currently */flows* API retrieves flow entities for all the users by default. > It is required to provide filter user i.e */flows?user=rohith* . This is > critical filter in secured environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6199) Support for listing flows with filter userid
[ https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041194#comment-16041194 ] Haibo Chen edited comment on YARN-6199 at 6/7/17 4:47 PM: -- Will follow up with a patch for branch YARN-5355 was (Author: haibochen): Will follow up with a patch for YARN-5355 > Support for listing flows with filter userid > > > Key: YARN-6199 > URL: https://issues.apache.org/jira/browse/YARN-6199 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Attachments: YARN-6199.00.patch, YARN-6199.01.patch > > > Currently */flows* API retrieves flow entities for all the users by default. > It is required to provide filter user i.e */flows?user=rohith* . This is > critical filter in secured environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6199) Support for listing flows with filter userid
[ https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040190#comment-16040190 ] Haibo Chen commented on YARN-6199: -- Thanks [~varun_saxena] for your review! I'll update the patch to address your comments. The daterange filter is already added as an optional query param of /flows. Maybe we should as a default daterange filter (say 7 days) to avoid full table scan? Agree with you that we should add a note in the documentation. Will incorporate this in the next patch. > Support for listing flows with filter userid > > > Key: YARN-6199 > URL: https://issues.apache.org/jira/browse/YARN-6199 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Attachments: YARN-6199.00.patch > > > Currently */flows* API retrieves flow entities for all the users by default. > It is required to provide filter user i.e */flows?user=rohith* . This is > critical filter in secured environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6199) Support for listing flows with filter userid
[ https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6199: - Attachment: YARN-6199-YARN-5355.01.patch > Support for listing flows with filter userid > > > Key: YARN-6199 > URL: https://issues.apache.org/jira/browse/YARN-6199 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Attachments: YARN-6199.00.patch, YARN-6199.01.patch, > YARN-6199-YARN-5355.01.patch > > > Currently */flows* API retrieves flow entities for all the users by default. > It is required to provide filter user i.e */flows?user=rohith* . This is > critical filter in secured environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041240#comment-16041240 ] Haibo Chen commented on YARN-6670: -- Compilation failed because of web ui. Will retrigger the jenkins job. > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch, > YARN-6670-YARN-1011.01.patch, YARN-6670-YARN-1011.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5094) some YARN container events have timestamp of -1 in REST output
[
https://issues.apache.org/jira/browse/YARN-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033262#comment-16033262
]
Haibo Chen commented on YARN-5094:
--
As indicated in the comments above, YARN-5169 is the real cause. I am planning
to fix the issue there.
> some YARN container events have timestamp of -1 in REST output
> --
>
> Key: YARN-5094
> URL: https://issues.apache.org/jira/browse/YARN-5094
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
> Labels: YARN-5355
> Attachments: YARN-5094-YARN-2928.001.patch
>
>
> Some events in the YARN container entities have timestamp of -1. The
> RM-generated container events have proper timestamps. It appears that it's
> the NM-generated events that have -1: YARN_CONTAINER_CREATED,
> YARN_CONTAINER_FINISHED, YARN_NM_CONTAINER_LOCALIZATION_FINISHED,
> YARN_NM_CONTAINER_LOCALIZATION_STARTED.
> In the YARN container page,
> {noformat}
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: -1,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: -1,
> info: { }
> }
> {noformat}
> I think the data itself is OK, but the values are not being populated in the
> REST output?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6047) Documentation updates for TimelineService v2
[ https://issues.apache.org/jira/browse/YARN-6047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033291#comment-16033291 ] Haibo Chen commented on YARN-6047: -- Linked YARN-6323 to document the first-time rolling upgrade case. > Documentation updates for TimelineService v2 > > > Key: YARN-6047 > URL: https://issues.apache.org/jira/browse/YARN-6047 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation, timelineserver >Reporter: Varun Saxena >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6047-YARN-5355.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-6670) Add over-allocation thresholds to allow per-node oversubscription
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen reopened YARN-6670: -- Reopen to add separate thresholds for CPU and memory over allocation > Add over-allocation thresholds to allow per-node oversubscription > - > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6677) Pause and preempt containers when root container cgroup goes over memory limit
Haibo Chen created YARN-6677: Summary: Pause and preempt containers when root container cgroup goes over memory limit Key: YARN-6677 URL: https://issues.apache.org/jira/browse/YARN-6677 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 3.0.0-alpha3 Reporter: Haibo Chen Assignee: Miklos Szegedi -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6677) Pause and preempt containers when root container cgroup goes over memory limit
[ https://issues.apache.org/jira/browse/YARN-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6677: - Issue Type: Sub-task (was: Bug) Parent: YARN-1011 > Pause and preempt containers when root container cgroup goes over memory limit > -- > > Key: YARN-6677 > URL: https://issues.apache.org/jira/browse/YARN-6677 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6670) Add separate configurations for cpu and memory oversubscription
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6670: - Summary: Add separate configurations for cpu and memory oversubscription (was: Add over-allocation thresholds to allow per-node oversubscription) > Add separate configurations for cpu and memory oversubscription > --- > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6670) Add over-allocation thresholds to allow per-node oversubscription
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen resolved YARN-6670. -- Resolution: Duplicate > Add over-allocation thresholds to allow per-node oversubscription > - > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6199) Support for listing flows with filter userid
[ https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6199: - Attachment: YARN-6199.00.patch Uploading a preliminary patch. I have also tested it with REST API. A known issue is that since it is using a regex string comparator, it gives results even when the specified user id is only a substring of what's in the table. Any suggestion to address this? > Support for listing flows with filter userid > > > Key: YARN-6199 > URL: https://issues.apache.org/jira/browse/YARN-6199 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Attachments: YARN-6199.00.patch > > > Currently */flows* API retrieves flow entities for all the users by default. > It is required to provide filter user i.e */flows?user=rohith* . This is > critical filter in secured environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039831#comment-16039831 ] Haibo Chen commented on YARN-6670: -- The patch is updated to address 1) and 2). In cases where the preemption threshold is lower than the allocation threshold, we could either assign a value larger than the allocation threshold to preemption threshold, or disable overallocation to avoid threshing. This patch chooses to disable overallocation, given it is hard to pick a certain value for preemption threshold that works well in all cases. > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch, > YARN-6670-YARN-1011.01.patch, YARN-6670-YARN-1011.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6670: - Attachment: YARN-6670-YARN-1011.02.patch > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch, > YARN-6670-YARN-1011.01.patch, YARN-6670-YARN-1011.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045570#comment-16045570 ] Haibo Chen commented on YARN-6670: -- [~miklos.szeg...@cloudera.com] NM_OVERALLOCATION_GENERAL_THRESHOLD applies to all resource types, including memory. Do you think the time period applies to memory? Can you please elaborate on the idea of time period as I not quite sure I understand? > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch, > YARN-6670-YARN-1011.01.patch, YARN-6670-YARN-1011.02.patch, > YARN-6670-YARN-1011.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045571#comment-16045571 ] Haibo Chen commented on YARN-6670: -- To move things forward, maybe we could continue the discussion of your time period idea in an improvement jira? > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch, > YARN-6670-YARN-1011.01.patch, YARN-6670-YARN-1011.02.patch, > YARN-6670-YARN-1011.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045568#comment-16045568 ] Haibo Chen commented on YARN-6670: -- Thanks [~kkaranasos] for the review! I will update the patch to address your comments, leaving the preemption threshold related one to be addressed in YARN-6705. > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch, > YARN-6670-YARN-1011.01.patch, YARN-6670-YARN-1011.02.patch, > YARN-6670-YARN-1011.03.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6705) Add separate NM preemption thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6705: - Issue Type: Sub-task (was: Task) Parent: YARN-1011 > Add separate NM preemption thresholds for cpu and memory > > > Key: YARN-6705 > URL: https://issues.apache.org/jira/browse/YARN-6705 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6670: - Attachment: YARN-6670-YARN-1011.04.patch > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch, > YARN-6670-YARN-1011.01.patch, YARN-6670-YARN-1011.02.patch, > YARN-6670-YARN-1011.03.patch, YARN-6670-YARN-1011.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6670: - Attachment: YARN-6670-YARN-1011.05.patch > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch, > YARN-6670-YARN-1011.01.patch, YARN-6670-YARN-1011.02.patch, > YARN-6670-YARN-1011.03.patch, YARN-6670-YARN-1011.04.patch, > YARN-6670-YARN-1011.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6199) Support for listing flows with filter userid
[ https://issues.apache.org/jira/browse/YARN-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6199: - Attachment: YARN-6199.02.patch Updated the patch against trunk to address the checkstyle issue. > Support for listing flows with filter userid > > > Key: YARN-6199 > URL: https://issues.apache.org/jira/browse/YARN-6199 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Attachments: YARN-6199.00.patch, YARN-6199.01.patch, > YARN-6199.02.patch, YARN-6199-YARN-5355.01.patch > > > Currently */flows* API retrieves flow entities for all the users by default. > It is required to provide filter user i.e */flows?user=rohith* . This is > critical filter in secured environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6563) ConcurrentModificationException in TimelineCollectorManager while stopping RM
[
https://issues.apache.org/jira/browse/YARN-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-6563:
-
Issue Type: Sub-task (was: Bug)
Parent: YARN-5355
> ConcurrentModificationException in TimelineCollectorManager while stopping RM
> -
>
> Key: YARN-6563
> URL: https://issues.apache.org/jira/browse/YARN-6563
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Haibo Chen
> Attachments: YARN-6563.00.patch
>
>
> It is seen that ConcurrentModificationException while stopping RM when ATSv2
> enabled.
> {noformat}
> 2017-05-05 15:04:11,563 WARN org.apache.hadoop.service.CompositeService: When
> stopping the service
> org.apache.hadoop.yarn.server.resourcemanager.timelineservice.RMTimelineCollectorManager
> : java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.serviceStop(TimelineCollectorManager.java:222)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1285)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6563) ConcurrentModificationException in TimelineCollectorManager while stopping RM
[
https://issues.apache.org/jira/browse/YARN-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-6563:
-
Attachment: YARN-6563.00.patch
> ConcurrentModificationException in TimelineCollectorManager while stopping RM
> -
>
> Key: YARN-6563
> URL: https://issues.apache.org/jira/browse/YARN-6563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Haibo Chen
> Attachments: YARN-6563.00.patch
>
>
> It is seen that ConcurrentModificationException while stopping RM when ATSv2
> enabled.
> {noformat}
> 2017-05-05 15:04:11,563 WARN org.apache.hadoop.service.CompositeService: When
> stopping the service
> org.apache.hadoop.yarn.server.resourcemanager.timelineservice.RMTimelineCollectorManager
> : java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.serviceStop(TimelineCollectorManager.java:222)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1285)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6559) Findbugs warning in YARN-5355 branch
[ https://issues.apache.org/jira/browse/YARN-6559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1640#comment-1640 ] Haibo Chen commented on YARN-6559: -- No idea why findbugs is still complaining about the warning that the patch is supposed to fix. Rekicking the build again. > Findbugs warning in YARN-5355 branch > > > Key: YARN-6559 > URL: https://issues.apache.org/jira/browse/YARN-6559 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Vrushali C >Priority: Minor > Attachments: FindBugs Report.html, YARN-6559-YARN-5355.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6457) Allow custom SSL configuration to be supplied in WebApps
[ https://issues.apache.org/jira/browse/YARN-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000107#comment-16000107 ] Haibo Chen commented on YARN-6457: -- [~PramodSSImmaneni], [~sanjaypujare], the approach looks great to me! Just one nit that not directly related to this issue, but would be nice to fix. We can move 'boolean needsClientAuth..' statement right before the return statement where it is used. Please create a new PR, and I'll commit it. > Allow custom SSL configuration to be supplied in WebApps > > > Key: YARN-6457 > URL: https://issues.apache.org/jira/browse/YARN-6457 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp, yarn >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare > Original Estimate: 96h > Remaining Estimate: 96h > > Currently a custom SSL store cannot be passed on to WebApps which forces the > embedded web-server to use the default keystore set up in ssl-server.xml for > the whole Hadoop cluster. There are cases where the Hadoop app needs to use > its own/custom keystore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6555) Enable flow context read (& corresponding write) for recovering application with NM restart
[
https://issues.apache.org/jira/browse/YARN-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1636#comment-1636
]
Haibo Chen commented on YARN-6555:
--
I remember vaguely that this is the issue that Li came across in YARN-6323
> Enable flow context read (& corresponding write) for recovering application
> with NM restart
>
>
> Key: YARN-6555
> URL: https://issues.apache.org/jira/browse/YARN-6555
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>
> If timeline service v2 is enabled and NM is restarted with recovery enabled,
> then NM fails to start and throws an error as "flow context can't be null".
> This is happening because the flow context did not exist before but now that
> timeline service v2 is enabled, ApplicationImpl expects it to exist.
> This would also happen even if flow context existed before but since we are
> not persisting it / reading it during
> ContainerManagerImpl#recoverApplication, it does not get passed in to
> ApplicationImpl.
> full stack trace
> {code}
> 2017-05-03 21:51:52,178 FATAL
> org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting
> NodeManager
> java.lang.IllegalArgumentException: flow context cannot be null
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:104)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:90)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverApplication(ContainerManagerImpl.java:318)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:280)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:267)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:276)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:588)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:649)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6563) ConcurrentModificationException in TimelineCollectorManager while stopping RM
[
https://issues.apache.org/jira/browse/YARN-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1628#comment-1628
]
Haibo Chen commented on YARN-6563:
--
Looks like we need to synchronize on the collectors object, per javadoc at
http://docs.oracle.com/javase/8/docs/api/java/util/Collections.html#synchronizedMap-java.util.Map-
> ConcurrentModificationException in TimelineCollectorManager while stopping RM
> -
>
> Key: YARN-6563
> URL: https://issues.apache.org/jira/browse/YARN-6563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Rohith Sharma K S
>
> It is seen that ConcurrentModificationException while stopping RM when ATSv2
> enabled.
> {noformat}
> 2017-05-05 15:04:11,563 WARN org.apache.hadoop.service.CompositeService: When
> stopping the service
> org.apache.hadoop.yarn.server.resourcemanager.timelineservice.RMTimelineCollectorManager
> : java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.serviceStop(TimelineCollectorManager.java:222)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1285)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6563) ConcurrentModificationException in TimelineCollectorManager while stopping RM
[
https://issues.apache.org/jira/browse/YARN-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1628#comment-1628
]
Haibo Chen edited comment on YARN-6563 at 5/7/17 7:57 PM:
--
Looks like we need to synchronize on the collectors object while iterating
over, per javadoc at
http://docs.oracle.com/javase/8/docs/api/java/util/Collections.html#synchronizedMap-java.util.Map-
was (Author: haibochen):
Looks like we need to synchronize on the collectors object, per javadoc at
http://docs.oracle.com/javase/8/docs/api/java/util/Collections.html#synchronizedMap-java.util.Map-
> ConcurrentModificationException in TimelineCollectorManager while stopping RM
> -
>
> Key: YARN-6563
> URL: https://issues.apache.org/jira/browse/YARN-6563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Rohith Sharma K S
>
> It is seen that ConcurrentModificationException while stopping RM when ATSv2
> enabled.
> {noformat}
> 2017-05-05 15:04:11,563 WARN org.apache.hadoop.service.CompositeService: When
> stopping the service
> org.apache.hadoop.yarn.server.resourcemanager.timelineservice.RMTimelineCollectorManager
> : java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.serviceStop(TimelineCollectorManager.java:222)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1285)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6563) ConcurrentModificationException in TimelineCollectorManager while stopping RM
[
https://issues.apache.org/jira/browse/YARN-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen reassigned YARN-6563:
Assignee: Haibo Chen
> ConcurrentModificationException in TimelineCollectorManager while stopping RM
> -
>
> Key: YARN-6563
> URL: https://issues.apache.org/jira/browse/YARN-6563
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Haibo Chen
>
> It is seen that ConcurrentModificationException while stopping RM when ATSv2
> enabled.
> {noformat}
> 2017-05-05 15:04:11,563 WARN org.apache.hadoop.service.CompositeService: When
> stopping the service
> org.apache.hadoop.yarn.server.resourcemanager.timelineservice.RMTimelineCollectorManager
> : java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.serviceStop(TimelineCollectorManager.java:222)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
> at
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1285)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5067) Support specifying resources for AM containers in SLS
[ https://issues.apache.org/jira/browse/YARN-5067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000239#comment-16000239 ] Haibo Chen commented on YARN-5067: -- Thanks [~yufeigu] for the patch! A few high-level comments. 1) Why make SLSConfiguration extend Configuration? The change seems unnecessary given that we can still use Configuration directly, unless I am missing something? 2) The patch does not apply cleanly now (a minor conflict) > Support specifying resources for AM containers in SLS > - > > Key: YARN-5067 > URL: https://issues.apache.org/jira/browse/YARN-5067 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Yufei Gu > Attachments: YARN-5067.001.patch, YARN-5067.002.patch > > > Now resource of application masters in SLS is hardcoded to mem=1024 vcores=1. > We should be able to specify AM resources from trace input file. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6457) Allow custom SSL configuration to be supplied in WebApps
[ https://issues.apache.org/jira/browse/YARN-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001333#comment-16001333 ] Haibo Chen commented on YARN-6457: -- If it is convenient for you, you can generate a patch (name it to YARN-6457.01.patch) and upload it here to kick off the precommit job. > Allow custom SSL configuration to be supplied in WebApps > > > Key: YARN-6457 > URL: https://issues.apache.org/jira/browse/YARN-6457 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp, yarn >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare > Attachments: YARN-6457.00.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > Currently a custom SSL store cannot be passed on to WebApps which forces the > embedded web-server to use the default keystore set up in ssl-server.xml for > the whole Hadoop cluster. There are cases where the Hadoop app needs to use > its own/custom keystore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6457) Allow custom SSL configuration to be supplied in WebApps
[ https://issues.apache.org/jira/browse/YARN-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001330#comment-16001330 ] Haibo Chen commented on YARN-6457: -- The findbugs warnings are not related. Mind fixing the javadoc issue? namely, 1) "@param sslConf the Configuration instance to use during loading of SSL conf" Now that we have renamed sslConf to conf, we need to rename it in javadoc as well. The description is not accurate either. How about "the Configuration instance to load custom SSL settings from"? 2) There is no @return in the method javadoc, let's add that. > Allow custom SSL configuration to be supplied in WebApps > > > Key: YARN-6457 > URL: https://issues.apache.org/jira/browse/YARN-6457 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp, yarn >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare > Attachments: YARN-6457.00.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > Currently a custom SSL store cannot be passed on to WebApps which forces the > embedded web-server to use the default keystore set up in ssl-server.xml for > the whole Hadoop cluster. There are cases where the Hadoop app needs to use > its own/custom keystore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6457) Allow custom SSL configuration to be supplied in WebApps
[ https://issues.apache.org/jira/browse/YARN-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6457: - Attachment: YARN-6457.00.patch Attaching a patch from the Pull Request to initiate pre-commit check on the change > Allow custom SSL configuration to be supplied in WebApps > > > Key: YARN-6457 > URL: https://issues.apache.org/jira/browse/YARN-6457 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp, yarn >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare > Attachments: YARN-6457.00.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > Currently a custom SSL store cannot be passed on to WebApps which forces the > embedded web-server to use the default keystore set up in ssl-server.xml for > the whole Hadoop cluster. There are cases where the Hadoop app needs to use > its own/custom keystore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6457) Allow custom SSL configuration to be supplied in WebApps
[ https://issues.apache.org/jira/browse/YARN-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001721#comment-16001721 ] Haibo Chen commented on YARN-6457: -- [~sanjaypujare], can you go to YARN-4562 to request a backport of it, if you want the fix in 2.7? I can do a backport of this patch afterwards. > Allow custom SSL configuration to be supplied in WebApps > > > Key: YARN-6457 > URL: https://issues.apache.org/jira/browse/YARN-6457 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp, yarn >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6457.00.patch, YARN-6457.01.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > Currently a custom SSL store cannot be passed on to WebApps which forces the > embedded web-server to use the default keystore set up in ssl-server.xml for > the whole Hadoop cluster. There are cases where the Hadoop app needs to use > its own/custom keystore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6457) Allow custom SSL configuration to be supplied in WebApps
[ https://issues.apache.org/jira/browse/YARN-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001667#comment-16001667 ] Haibo Chen commented on YARN-6457: -- Patch LGTM+1, will fix the minor checkstyle issue when I commit. > Allow custom SSL configuration to be supplied in WebApps > > > Key: YARN-6457 > URL: https://issues.apache.org/jira/browse/YARN-6457 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp, yarn >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare > Attachments: YARN-6457.00.patch, YARN-6457.01.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > Currently a custom SSL store cannot be passed on to WebApps which forces the > embedded web-server to use the default keystore set up in ssl-server.xml for > the whole Hadoop cluster. There are cases where the Hadoop app needs to use > its own/custom keystore. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6657) The column family indicated in AppToFlowTable class javadoc is wrong
[ https://issues.apache.org/jira/browse/YARN-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029768#comment-16029768 ] Haibo Chen commented on YARN-6657: -- Oh, I did not realize that. I guess in this case, no need to get this in trunk. > The column family indicated in AppToFlowTable class javadoc is wrong > > > Key: YARN-6657 > URL: https://issues.apache.org/jira/browse/YARN-6657 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Trivial > Attachments: YARN-6657.00.patch > > > AppToFlowTable javadoc says the column family is info, whereas in reality it > is mapping. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6628) Unexpected jackson-core-2.2.3 dependency introduced
[
https://issues.apache.org/jira/browse/YARN-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018113#comment-16018113
]
Haibo Chen commented on YARN-6628:
--
Based on its pom file, fst depends on jackson-core:2.8.8
{code:xml}
com.fasterxml.jackson.core
jackson-core
2.8.8
{code}
It is overridden by the version specified in hadoop-2.8, which is 2.2.3. That's
why jackson-core-2.2.3.jar is in share/hadoop/yarn/lib
{code:xml}
\- de.ruedigermoeller:fst:jar:2.50:compile
+- com.fasterxml.jackson.core:jackson-core:jar:2.2.3:compile (version
managed from 2.8.8)
\- com.cedarsoftware:java-util:jar:1.9.0:compile
\- com.cedarsoftware:json-io:jar:2.5.1:compile
{code}
Strangely enough, according to mvn dependency:tree, jackson-core is not
dependent on by fst in trunk
{code}
\- de.ruedigermoeller:fst:jar:2.50:compile
\- com.cedarsoftware:java-util:jar:1.9.0:compile
\- com.cedarsoftware:json-io:jar:2.5.1:compile
{code}
and jackson-core.jar is NOT in share/hadoop/yarn/lib/ even though
hadoop-yarn-common and other modules depend on it.
{code}
+- org.apache.hadoop:hadoop-yarn-common:jar:3.0.0-alpha3-SNAPSHOT:compile
+- com.fasterxml.jackson.core:jackson-core:jar:2.7.8:compile
+-
com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.7.8:compile
\- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.7.8:compile
\- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.7.8:compile
{code}
Any idea how this could have happened?
> Unexpected jackson-core-2.2.3 dependency introduced
> ---
>
> Key: YARN-6628
> URL: https://issues.apache.org/jira/browse/YARN-6628
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 2.8.1
>Reporter: Jason Lowe
>Priority: Blocker
>
> The change in YARN-5894 caused jackson-core-2.2.3.jar to be added in
> share/hadoop/yarn/lib/. This added dependency seems to be incompatible with
> jackson-core-asl-1.9.13.jar which is also shipped as a dependency. This new
> jackson-core jar ends up breaking jobs that ran fine on 2.8.0.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6628) Unexpected jackson-core-2.2.3 dependency introduced
[ https://issues.apache.org/jira/browse/YARN-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018133#comment-16018133 ] Haibo Chen commented on YARN-6628: -- Similar for jackson-core-asl, it shows up in the dependency tree in both trunk and branch-2.8, but the jar only shows up in 2.8 branch > Unexpected jackson-core-2.2.3 dependency introduced > --- > > Key: YARN-6628 > URL: https://issues.apache.org/jira/browse/YARN-6628 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 2.8.1 >Reporter: Jason Lowe >Priority: Blocker > > The change in YARN-5894 caused jackson-core-2.2.3.jar to be added in > share/hadoop/yarn/lib/. This added dependency seems to be incompatible with > jackson-core-asl-1.9.13.jar which is also shipped as a dependency. This new > jackson-core jar ends up breaking jobs that ran fine on 2.8.0. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6670) Add a new configuration, over-allocation threshold, to allow per-node oversubscription
Haibo Chen created YARN-6670: Summary: Add a new configuration, over-allocation threshold, to allow per-node oversubscription Key: YARN-6670 URL: https://issues.apache.org/jira/browse/YARN-6670 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 3.0.0-alpha3 Reporter: Haibo Chen Assignee: Haibo Chen -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6323) Rolling upgrade/config change is broken on timeline v2.
[ https://issues.apache.org/jira/browse/YARN-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032185#comment-16032185 ] Haibo Chen commented on YARN-6323: -- In rolling upgrade cases where the newer version of YARN cluster has ATSv2 enabled, I think we should try to keep the default flow context for all applications that did not specify flow information consistent across all NMs and RM. Otherwise, RM will think application A is in flow X, whereas NMs would think application A is in flow B, and the result in HBase can be confusing. I wonder how much value it adds for RM to assign the default flow context based on application name (in case application name is absent, it falls back to application id). If that does not matter much, I'd propose both RM and NMs assign default flow context based on application id. > Rolling upgrade/config change is broken on timeline v2. > > > Key: YARN-6323 > URL: https://issues.apache.org/jira/browse/YARN-6323 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Attachments: YARN-6323.001.patch > > > Found this issue when deploying on real clusters. If there are apps running > when we enable timeline v2 (with work preserving restart enabled), node > managers will fail to start due to missing app context data. We should > probably assign some default names to these "left over" apps. I believe it's > suboptimal to let users clean up the whole cluster before enabling timeline > v2. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6670: - Attachment: YARN-6670-YARN-1011.01.patch > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch, > YARN-6670-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6690) Consolidate NM overallocation thresholds with ResourceTypes
[ https://issues.apache.org/jira/browse/YARN-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6690: - Issue Type: Sub-task (was: Improvement) Parent: YARN-1011 > Consolidate NM overallocation thresholds with ResourceTypes > > > Key: YARN-6690 > URL: https://issues.apache.org/jira/browse/YARN-6690 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > > YARN-3926 (ResourceTypes) introduces a new class ResourceInformation to > encapsulate all information about a given resource type (e.g. type, value, > unit). We could add the overallocation thresholds to it as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6690) Consolidate NM overallocation thresholds with ResourceTypes
Haibo Chen created YARN-6690: Summary: Consolidate NM overallocation thresholds with ResourceTypes Key: YARN-6690 URL: https://issues.apache.org/jira/browse/YARN-6690 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 3.0.0-alpha3 Reporter: Haibo Chen Assignee: Haibo Chen YARN-3926 (ResourceTypes) introduces a new class ResourceInformation to encapsulate all information about a given resource type (e.g. type, value, unit). We could add the overallocation thresholds to it as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6670) Add separate NM overallocation thresholds for cpu and memory
[ https://issues.apache.org/jira/browse/YARN-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037483#comment-16037483 ] Haibo Chen commented on YARN-6670: -- Thanks [~kasha] for your review ! In ResourceTypes, a new class, ResourceInformation, is added to encapsulate all information about a given resource type (e.g. type, value, unit), so I suppose you mean we could add the oversubscription threshold of each resource type to it as well? If so, I will create another jira to consolidate given that YARN-3926 is not merged yet. Will follow up with a new patch to address your comment 1) > Add separate NM overallocation thresholds for cpu and memory > > > Key: YARN-6670 > URL: https://issues.apache.org/jira/browse/YARN-6670 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: YARN-6670-YARN-1011.00.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
