[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527332#comment-14527332 ] Xuan Gong commented on YARN-2048: - After YARN-1809, from RM web ui, we could find all the running containers. And from ATS Web ui, we could find all the completed containers. List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527334#comment-14527334 ] Xuan Gong commented on YARN-2048: - Close this as fixed List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996122#comment-13996122 ] Wangda Tan commented on YARN-2048: -- Thanks [~zjshen] and [~coderplay] for this explanation. Now I can understand contexts of them. List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996096#comment-13996096 ] Zhijie Shen commented on YARN-2048: --- bq. Seems Zhijie Shen's patch fetch containers from ApplicationContext. Currently, the history web UI fetches data (app/attempt/container) from ApplicationContext, while RM web UI does it from RM context. My ultimate goal to uniform both history and rm web UI, and uniform the data source with the RPC protocol. List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995553#comment-13995553 ] Tsuyoshi OZAWA commented on YARN-2048: -- +1 for the idea. Looking forward. List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Reporter: Min Zhou Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996577#comment-13996577 ] Zhijie Shen commented on YARN-2048: --- bq. the only implementation of ApplcationContext is ApplicationHistoryManagerImpl, which retrieves containers information from history store. It's a historic problem of history service development. We plan to have a common interface for both RM and history server to retrieve the app information. bq. How do you fetch the containers info from a historyserver and display it on the RM web? We are supposed to provide the consistent RM and history web UI, but RM web UI shows the running and cached completed apps, while history web UI shows the completed apps. bq. If the information is from history store, seems RM won't get that kind of info until the application is done? Sometimes user's application might be a long-live application, never finish unless user kill it. Not exactly. RM web UI will always show the running app information. Currently, history web UI will only show the app information once the app is completed due to the current history store implementation. After we rebase the store on top of timeline store, we should get rid of the issue. bq. Seems the only way providing containers info to RM is to maintain a list in RMAppAttempImpl, which was my way as well. Months ago, I define ApplcationContext to be common interface for retrieve the app information, including container. Recently, released in 2.4.0, we have already supported the analog RPC interfaces in both RM and history server to retrieve app information, again including getContainer(s). This is reason why in YARN-1809, I'd like to rebase both RM and history web UI to retrieve the app information from the analog RPC interfaces. In this way, both RM and history web UI are showing consistent app information via both CLI and web pages. Of course, we'd like to make REST APIs uniformed as well. List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996083#comment-13996083 ] Min Zhou commented on YARN-2048: [~zjshen] hmm... from your patch, it's indeed a duplicate. Please go ahead. List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996055#comment-13996055 ] Zhijie Shen commented on YARN-2048: --- Is this duplicate of YARN-1809? In YARN-1809, one change is to make RM web list containers as well. List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995824#comment-13995824 ] Wangda Tan commented on YARN-2048: -- Hi [~coderplay], +1 for this idea, it should be very helpful to debug YARN applications. I took a look at your patch, some comments, 1) When AM restarted, all containers will be copied to new attempt's container list, which might lead user confuse why new attempt has all containers from old attempts 2) You might need consider following JIRAs to make app-containers page included all expected containers, YARN-556,YARN-1885,YARN-1489 List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996126#comment-13996126 ] Min Zhou commented on YARN-2048: [~zjshen] Currently, the only implementation of ApplcationContext is ApplicationHistoryManagerImpl, which retrieves containers information from history store. Quesions: # How do you fetch the containers info from a historyserver and display it on the RM web? # If the information is from history store, seems RM won't get that kind of info until the application is done? Sometimes user's application might be a long-live application, never finish unless user kill it. # Seems the only way providing containers info to RM is to maintain a list in RMAppAttempImpl, which was my way as well. Min List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996062#comment-13996062 ] Wangda Tan commented on YARN-2048: -- Hi [~zjshen], I think these two JIRAs covers similar issues. I took a quick look at your patch, I found you haven't changed RMAppAttempt/RMApp/RMContainer, so could you please elaborate a little about what you did to get containers from an application? Ask this because I'm thinking [~coderplay]'s patch can be considered as complementary of your solution if you haven't implemented it. List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995863#comment-13995863 ] Min Zhou commented on YARN-2048: Thanks [~leftnoteasy], I will take a look on the those stuff you mentioned and resubmit another one. List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996752#comment-13996752 ] Min Zhou commented on YARN-2048: Make sense now, please go ahead. List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997089#comment-13997089 ] Zhijie Shen commented on YARN-2048: --- Hi [~coderplay], what is the exact requirement? 2.4 has been already out, and YARN-1809 was not able to get in as we didn't have enough time test it thoroughly. Hence we target 2.5 for that jira, which should cover the scenario here. Usually we won't include major change in a maintain release, such as 2.4.1. However, if you think it is really urgent and required fix for 2.4.1, please feel free to reopen it, and set target version to 2.4.1. Thanks! List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2048) List all of the containers of an application from the yarn web
[ https://issues.apache.org/jira/browse/YARN-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996089#comment-13996089 ] Min Zhou commented on YARN-2048: [~leftnoteasy] Seems [~zjshen]'s patch fetch containers from ApplicationContext. I will confirm it after downloading a copy of yarn's code to my laptop. List all of the containers of an application from the yarn web -- Key: YARN-2048 URL: https://issues.apache.org/jira/browse/YARN-2048 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager, webapp Affects Versions: 2.3.0, 2.4.0, 2.5.0 Reporter: Min Zhou Attachments: YARN-2048-trunk-v1.patch Currently, Yarn haven't provide a way to list all of the containers of an application from its web. This kind of information is needed by the application user. They can conveniently know how many containers their applications already acquired as well as which nodes those containers were launched on. They also want to view the logs of each container of an application. One approach is maintain a container list in RMAppImpl and expose this info to Application page. I will submit a patch soon -- This message was sent by Atlassian JIRA (v6.2#6252)
