[
https://issues.apache.org/jira/browse/YARN-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923208#comment-15923208
]
yunjiong zhao commented on YARN-6285:
-------------------------------------
Sorry for late response.
2.25 seconds in getApplications doesn't include ResourceRequest.
Most of the time was spend on getLogAggregationReportsForApp as stack trace
shows.
I believe below code change should improve the performance (will test later)
{code}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
index ab283e7..926c757 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
@@ -296,6 +296,8 @@ public static ReservationRequestInterpreter
convertFromProtoFormat(
* Log Aggregation Status
*/
private static final String LOG_AGGREGATION_STATUS_PREFIX = "LOG_";
+ private static final int LOG_AGGREGATION_STATUS_PREFIX_LEN =
+ LOG_AGGREGATION_STATUS_PREFIX.length();
public static LogAggregationStatusProto convertToProtoFormat(
LogAggregationStatus e) {
return LogAggregationStatusProto.valueOf(LOG_AGGREGATION_STATUS_PREFIX
@@ -304,8 +306,8 @@ public static LogAggregationStatusProto
convertToProtoFormat(
public static LogAggregationStatus convertFromProtoFormat(
LogAggregationStatusProto e) {
- return LogAggregationStatus.valueOf(e.name().replace(
- LOG_AGGREGATION_STATUS_PREFIX, ""));
+ return LogAggregationStatus.valueOf(e.name().substring(
+ LOG_AGGREGATION_STATUS_PREFIX_LEN));
}
/*
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
index 9f00b2e..1db66a5 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
@@ -1700,17 +1700,16 @@ public ResourceRequest getAMResourceRequest() {
Map<NodeId, LogAggregationReport> outputs =
new HashMap<NodeId, LogAggregationReport>();
outputs.putAll(logAggregationStatus);
- if (!isLogAggregationFinished()) {
+ if (!isLogAggregationFinished() && isAppInFinalState(this) &&
+ System.currentTimeMillis() > this.logAggregationStartTime
+ + this.logAggregationStatusTimeout) {
for (Entry<NodeId, LogAggregationReport> output : outputs.entrySet()) {
if (!output.getValue().getLogAggregationStatus()
.equals(LogAggregationStatus.TIME_OUT)
&& !output.getValue().getLogAggregationStatus()
.equals(LogAggregationStatus.SUCCEEDED)
&& !output.getValue().getLogAggregationStatus()
- .equals(LogAggregationStatus.FAILED)
- && isAppInFinalState(this)
- && System.currentTimeMillis() > this.logAggregationStartTime
- + this.logAggregationStatusTimeout) {
+ .equals(LogAggregationStatus.FAILED)) {
output.getValue().setLogAggregationStatus(
LogAggregationStatus.TIME_OUT);
}
{code}
Should I open a new issue for those changes?
{quote}
1) Add parameter to indicate if we should include
ResourceRequest/getLogAggregationReportsForApp in the response, default is true
to make it compatible. (Can be done if above experimental shows it really
helps).
{quote}
This will help if user use those parameters.
> Add option to set max limit on ResourceManager for
> ApplicationClientProtocol.getApplications
> --------------------------------------------------------------------------------------------
>
> Key: YARN-6285
> URL: https://issues.apache.org/jira/browse/YARN-6285
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: yunjiong zhao
> Assignee: yunjiong zhao
> Attachments: YARN-6285.001.patch, YARN-6285.002.patch,
> YARN-6285.003.patch
>
>
> When users called ApplicationClientProtocol.getApplications, it will return
> lots of data, and generate lots of garbage on ResourceManager which caused
> long time GC.
> For example, on one of our RM, when called rest API " http://<rm http
> address:port>/ws/v1/cluster/apps" it can return 150MB data which have 944
> applications.
> getApplications have limit parameter, but some user might not set it, and
> then the limit will be Long.MAX_VALUE.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
