[hadoop] branch branch-3.2 updated: Make upstream aware of 3.2.1 release.
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.2 by this push: new f9f0338 Make upstream aware of 3.2.1 release. f9f0338 is described below commit f9f0338104e81f1f67350083557dbccc9bd1bd80 Author: Rohith Sharma K S AuthorDate: Mon Sep 23 06:20:54 2019 + Make upstream aware of 3.2.1 release. --- .../site/markdown/release/3.2.1/CHANGELOG.3.2.1.md | 553 + .../markdown/release/3.2.1/RELEASENOTES.3.2.1.md | 80 +++ .../dev-support/jdiff/Apache_Hadoop_HDFS_3.2.1.xml | 674 + hadoop-project-dist/pom.xml| 2 +- 4 files changed, 1308 insertions(+), 1 deletion(-) diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/CHANGELOG.3.2.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/CHANGELOG.3.2.1.md new file mode 100644 index 000..64e249e --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/CHANGELOG.3.2.1.md @@ -0,0 +1,553 @@ + + +# Apache Hadoop Changelog + +## Release 3.2.1 - 2019-09-10 + +### INCOMPATIBLE CHANGES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-15922](https://issues.apache.org/jira/browse/HADOOP-15922) | DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL | Major | common, kms | He Xiaoqiao | He Xiaoqiao | + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-15950](https://issues.apache.org/jira/browse/HADOOP-15950) | Failover for LdapGroupsMapping | Major | common, security | Lukas Majercak | Lukas Majercak | +| [YARN-7055](https://issues.apache.org/jira/browse/YARN-7055) | YARN Timeline Service v.2: beta 1 / GA | Major | timelineclient, timelinereader, timelineserver | Vrushali C | | +| [YARN-9761](https://issues.apache.org/jira/browse/YARN-9761) | Allow overriding application submissions based on server side configs | Major | . | Jonathan Hung | pralabhkumar | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-15676](https://issues.apache.org/jira/browse/HADOOP-15676) | Cleanup TestSSLHttpServer | Minor | common | Szilard Nemeth | Szilard Nemeth | +| [YARN-8896](https://issues.apache.org/jira/browse/YARN-8896) | Limit the maximum number of container assignments per heartbeat | Major | . | Weiwei Yang | Zhankun Tang | +| [YARN-8618](https://issues.apache.org/jira/browse/YARN-8618) | Yarn Service: When all the components of a service have restart policy NEVER then initiation of service upgrade should fail | Major | . | Chandni Singh | Chandni Singh | +| [HADOOP-15804](https://issues.apache.org/jira/browse/HADOOP-15804) | upgrade to commons-compress 1.18 | Major | . | PJ Fanning | Akira Ajisaka | +| [YARN-8916](https://issues.apache.org/jira/browse/YARN-8916) | Define a constant "docker" string in "ContainerRuntimeConstants.java" for better maintainability | Minor | . | Zhankun Tang | Zhankun Tang | +| [YARN-8908](https://issues.apache.org/jira/browse/YARN-8908) | Fix errors in yarn-default.xml related to GPU/FPGA | Major | . | Zhankun Tang | Zhankun Tang | +| [HDFS-13941](https://issues.apache.org/jira/browse/HDFS-13941) | make storageId in BlockPoolTokenSecretManager.checkAccess optional | Major | . | Ajay Kumar | Ajay Kumar | +| [HDFS-14029](https://issues.apache.org/jira/browse/HDFS-14029) | Sleep in TestLazyPersistFiles should be put into a loop | Trivial | hdfs | Adam Antal | Adam Antal | +| [YARN-8915](https://issues.apache.org/jira/browse/YARN-8915) | Update the doc about the default value of "maximum-container-assignments" for capacity scheduler | Minor | . | Zhankun Tang | Zhankun Tang | +| [HADOOP-15855](https://issues.apache.org/jira/browse/HADOOP-15855) | Review hadoop credential doc, including object store details | Minor | documentation, security | Steve Loughran | Steve Loughran | +| [YARN-7225](https://issues.apache.org/jira/browse/YARN-7225) | Add queue and partition info to RM audit log | Major | resourcemanager | Jonathan Hung | Eric Payne | +| [HADOOP-15687](https://issues.apache.org/jira/browse/HADOOP-15687) | Credentials class should allow access to aliases | Trivial | . | Lars Francke | Lars Francke | +| [YARN-8969](https://issues.apache.org/jira/browse/YARN-8969) | AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting | Major | . | Wanqiang Ji | Wanqiang Ji | +| [YARN-8977](https://issues.apache.org/jira/browse/YARN-8977) | Remove unnecessary type casting when calling AbstractYarnSchedule
[hadoop] branch trunk updated: Make upstream aware of 3.2.1 release.
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 4c0a7a9 Make upstream aware of 3.2.1 release. 4c0a7a9 is described below commit 4c0a7a9e13c68b24c94009927f9fd3987abd8144 Author: Rohith Sharma K S AuthorDate: Mon Sep 23 06:20:54 2019 + Make upstream aware of 3.2.1 release. --- .../site/markdown/release/3.2.1/CHANGELOG.3.2.1.md | 553 + .../markdown/release/3.2.1/RELEASENOTES.3.2.1.md | 80 +++ .../dev-support/jdiff/Apache_Hadoop_HDFS_3.2.1.xml | 674 + hadoop-project-dist/pom.xml| 2 +- 4 files changed, 1308 insertions(+), 1 deletion(-) diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/CHANGELOG.3.2.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/CHANGELOG.3.2.1.md new file mode 100644 index 000..64e249e --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.1/CHANGELOG.3.2.1.md @@ -0,0 +1,553 @@ + + +# Apache Hadoop Changelog + +## Release 3.2.1 - 2019-09-10 + +### INCOMPATIBLE CHANGES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-15922](https://issues.apache.org/jira/browse/HADOOP-15922) | DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL | Major | common, kms | He Xiaoqiao | He Xiaoqiao | + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-15950](https://issues.apache.org/jira/browse/HADOOP-15950) | Failover for LdapGroupsMapping | Major | common, security | Lukas Majercak | Lukas Majercak | +| [YARN-7055](https://issues.apache.org/jira/browse/YARN-7055) | YARN Timeline Service v.2: beta 1 / GA | Major | timelineclient, timelinereader, timelineserver | Vrushali C | | +| [YARN-9761](https://issues.apache.org/jira/browse/YARN-9761) | Allow overriding application submissions based on server side configs | Major | . | Jonathan Hung | pralabhkumar | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|: |: | :--- |: |: |: | +| [HADOOP-15676](https://issues.apache.org/jira/browse/HADOOP-15676) | Cleanup TestSSLHttpServer | Minor | common | Szilard Nemeth | Szilard Nemeth | +| [YARN-8896](https://issues.apache.org/jira/browse/YARN-8896) | Limit the maximum number of container assignments per heartbeat | Major | . | Weiwei Yang | Zhankun Tang | +| [YARN-8618](https://issues.apache.org/jira/browse/YARN-8618) | Yarn Service: When all the components of a service have restart policy NEVER then initiation of service upgrade should fail | Major | . | Chandni Singh | Chandni Singh | +| [HADOOP-15804](https://issues.apache.org/jira/browse/HADOOP-15804) | upgrade to commons-compress 1.18 | Major | . | PJ Fanning | Akira Ajisaka | +| [YARN-8916](https://issues.apache.org/jira/browse/YARN-8916) | Define a constant "docker" string in "ContainerRuntimeConstants.java" for better maintainability | Minor | . | Zhankun Tang | Zhankun Tang | +| [YARN-8908](https://issues.apache.org/jira/browse/YARN-8908) | Fix errors in yarn-default.xml related to GPU/FPGA | Major | . | Zhankun Tang | Zhankun Tang | +| [HDFS-13941](https://issues.apache.org/jira/browse/HDFS-13941) | make storageId in BlockPoolTokenSecretManager.checkAccess optional | Major | . | Ajay Kumar | Ajay Kumar | +| [HDFS-14029](https://issues.apache.org/jira/browse/HDFS-14029) | Sleep in TestLazyPersistFiles should be put into a loop | Trivial | hdfs | Adam Antal | Adam Antal | +| [YARN-8915](https://issues.apache.org/jira/browse/YARN-8915) | Update the doc about the default value of "maximum-container-assignments" for capacity scheduler | Minor | . | Zhankun Tang | Zhankun Tang | +| [HADOOP-15855](https://issues.apache.org/jira/browse/HADOOP-15855) | Review hadoop credential doc, including object store details | Minor | documentation, security | Steve Loughran | Steve Loughran | +| [YARN-7225](https://issues.apache.org/jira/browse/YARN-7225) | Add queue and partition info to RM audit log | Major | resourcemanager | Jonathan Hung | Eric Payne | +| [HADOOP-15687](https://issues.apache.org/jira/browse/HADOOP-15687) | Credentials class should allow access to aliases | Trivial | . | Lars Francke | Lars Francke | +| [YARN-8969](https://issues.apache.org/jira/browse/YARN-8969) | AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting | Major | . | Wanqiang Ji | Wanqiang Ji | +| [YARN-8977](https://issues.apache.org/jira/browse/YARN-8977) | Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode
[hadoop] annotated tag rel/release-3.2.1 created (now 191cfa1)
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a change to annotated tag rel/release-3.2.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git. at 191cfa1 (tag) tagging b3cbbb467e22ea829b3808f4b7b01d07e0bf3842 (commit) by Rohith Sharma K S on Mon Sep 23 04:26:08 2019 + - Log - Hadoop 3.2.1 release -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAABAgAGBQJdiElgAAoJEKw9ElM8ygIx9gYQAI+gvfT6pUvwtuOWIL09veaZ rzEUb/FhIu/sdkldWG/dL8Qn4R7/FLjhQxovEQM3e/L64IxeLyXpZE0/+IrSgtVA fpcKHnSleaGOx15Epph8vtDKtWM3bkvOaN/aaiNf9wyMBYFCphTpIbTIaO4sO9e5 qtEJ0v0/IQw08jkQyUvFMle6bmfn6ty8iIHqPCx2ZRlgSpipm19QBJTP2ktEfRRV TIkfD6mYDPyv/N85L6FP04wY5TDstFnikIkOCVFINs/4emJCuWmJ5HlS8tmnuaSK C/nTX2efHvWyJsHsCc+GciOSJ94/uZHjWN05BhtaoWd/dPvmriWGjBq9XUnZiHZs RgVGJV042JshUVaxP8IXEksEILbKc8kKGB2NOeTnHUgVcCx8bCVahqTv6eRFctDi bLv4duk9hOVTUNaPeBbPxLwtVDLmL9P9llUvullpynd66KUhxwSvwfODM1orLsNu D361NFEPJQUn+cvCyU0VYxjA9Pfrq4grmKcQz0XXWV9qdQiDsHEzSpUSmFOXoBTz DtsG1AnGZPe4LCy3PzcCvJrItSsPfY7p3sNIUu95C0CL+0CJmVPomMhyjLcu5Jvd goi1dUuHrnWI0bIp1+/DEuoCUKVIsduCrRgTOdz/DrGdYqabhMrLKUzD9FMIhOLU o/RlHme8U3pelhWY6cTO =CB7c -END PGP SIGNATURE- --- No new revisions were added by this update. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] annotated tag release-3.2.1-RC0 created (now fa95695)
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a change to annotated tag release-3.2.1-RC0 in repository https://gitbox.apache.org/repos/asf/hadoop.git. at fa95695 (tag) tagging b3cbbb467e22ea829b3808f4b7b01d07e0bf3842 (commit) by Rohith Sharma K S on Wed Sep 11 05:11:54 2019 + - Log - Release candidate - 3.2.1-RC0 -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAABAgAGBQJdeIIaAAoJEKw9ElM8ygIxyDQQAKHZIGmqexBZ13KC9TsB5GRB DgFZpQOZl5UQduaJiVrb4cKDoFcvbwBqc1RXyJer5lNwHVq6EwtrjM2dlZq4pqqd rOZavqAQSUYfwQsSz4k4A8UbJQtBRCKYpu0T+qAqIHODNyRm0CMQw7g//GYdYLxl e1VlymyEhYRe05Shd/sBO0o3KWO0uf83HCb1EgI6yIUJWroL4fDXX0y3g0bp7ZMH 1wNMch+rS+kJLjZRqBzjY/gVWweB5WfZDvm/UjI7iI9bqjeRe/OeNNAek7y4Ah/p BE2ityy0bDb84fp3uUyypItEYyMaJc9OUbtRq00IQZ/mNrEd8Kl3pERXifIwmeEi q+JR5PgrosHWL3iYHFKuIEHEPzDQhJuIs4lYgQutcu9GYfx3Syq0ce1rCD13jDe8 CWeYiF0b0KgrQdNUdQvnQMPH580PNVFqhu+J3a2X5z57kz/5MNohlseSa3qHX65G PbszD8TeFxuA1idY1jUEseSqXppWA4CDJim/r2kBeAtamdvC1eXuPKYFVl5nsrc8 nizEEKc5gMOYkJ2ewbIM16Dxws4p2Q31ZxzGGWXzXKT8Nthx3GCvUBCKAR4XkIbX 1f8kcYF/MGIFmmzh0gLkcpeJG2lFquZwMcoisoyFN/ducGovE5C81XCdEZnKUFua mWAB8YDN63qzIJm+jRaA =w0DT -END PGP SIGNATURE- --- No new revisions were added by this update. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2.1 updated (865b49a -> b3cbbb4)
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a change to branch branch-3.2.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 865b49a Preparing for 3.2.1 release. new 46a1a6f YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph new b3cbbb4 YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase is Down. Contributed by Prabhu Joseph. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../resourcemanager/recovery/RMStateStore.java | 7 ++ .../recovery/RMStateUpdateAppEvent.java| 7 +- .../server/resourcemanager/rmapp/RMAppImpl.java| 2 +- .../rmapp/TestRMAppTransitions.java| 7 +- .../storage/HBaseTimelineWriterImpl.java | 74 -- 5 files changed, 62 insertions(+), 35 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 01/02: YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 46a1a6fc7d5e18f0caec48ce33ecb21901adfac8
Author: Jonathan Hung
AuthorDate: Mon Sep 9 00:23:04 2019 -0700
YARN-9820. RM logs InvalidStateTransitionException when app is submitted.
Contributed by Prabhu Joseph
---
.../hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java | 7 +++
.../server/resourcemanager/recovery/RMStateUpdateAppEvent.java | 7 ++-
.../apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java | 2 +-
.../yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java| 7 ++-
4 files changed, 20 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
index ccd6fc9..01606bc 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
@@ -868,6 +868,13 @@ public abstract class RMStateStore extends AbstractService
{
getRMStateStoreEventHandler().handle(new RMStateUpdateAppEvent(appState));
}
+ @SuppressWarnings("unchecked")
+ public void updateApplicationState(ApplicationStateData appState,
+ boolean notifyApp) {
+getRMStateStoreEventHandler().handle(new RMStateUpdateAppEvent(appState,
+notifyApp));
+ }
+
public void updateApplicationStateSynchronously(ApplicationStateData
appState,
boolean notifyApp, SettableFuture resultFuture) {
handleStoreEvent(
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateUpdateAppEvent.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateUpdateAppEvent.java
index 0a6220b..a43d43e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateUpdateAppEvent.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateUpdateAppEvent.java
@@ -30,9 +30,14 @@ public class RMStateUpdateAppEvent extends RMStateStoreEvent
{
private SettableFuture future;
public RMStateUpdateAppEvent(ApplicationStateData appState) {
+this (appState, true);
+ }
+
+ public RMStateUpdateAppEvent(ApplicationStateData appState,
+ boolean notifyApplication) {
super(RMStateStoreEventType.UPDATE_APP);
this.appState = appState;
-this.notifyApplication = true;
+this.notifyApplication = notifyApplication;
this.future = null;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
index a1e4558..84c1d86 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
@@ -1037,7 +1037,7 @@ public class RMAppImpl implements RMApp, Recoverable {
app.callerContext);
appState.setApplicationTimeouts(app.getApplicationTimeouts());
appState.setLaunchTime(event.getTimestamp());
-app.rmContext.getStateStore().updateApplicationState(appState);
+app.rmContext.getStateStore().updateApplicationState(appState, false);
app.launchTime = event.getTimestamp();
}
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
index f5dcefb..b32a
[hadoop] 02/02: YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase is Down. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit b3cbbb467e22ea829b3808f4b7b01d07e0bf3842
Author: Abhishek Modi
AuthorDate: Mon Sep 9 14:26:00 2019 +0530
YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase is Down.
Contributed by Prabhu Joseph.
---
.../storage/HBaseTimelineWriterImpl.java | 74 --
1 file changed, 42 insertions(+), 32 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineWriterImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineWriterImpl.java
index a398feb..dda004d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineWriterImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineWriterImpl.java
@@ -630,39 +630,49 @@ public class HBaseTimelineWriterImpl extends
AbstractService implements
*/
@Override
protected void serviceStop() throws Exception {
-if (entityTable != null) {
- LOG.info("closing the entity table");
- // The close API performs flushing and releases any resources held
- entityTable.close();
+boolean isStorageUp = true;
+try {
+ storageMonitor.checkStorageIsUp();
+} catch (IOException e) {
+ LOG.warn("Failed to close the timeline tables as Hbase is down", e);
+ isStorageUp = false;
}
-if (appToFlowTable != null) {
- LOG.info("closing the app_flow table");
- // The close API performs flushing and releases any resources held
- appToFlowTable.close();
-}
-if (applicationTable != null) {
- LOG.info("closing the application table");
- applicationTable.close();
-}
-if (flowRunTable != null) {
- LOG.info("closing the flow run table");
- // The close API performs flushing and releases any resources held
- flowRunTable.close();
-}
-if (flowActivityTable != null) {
- LOG.info("closing the flowActivityTable table");
- // The close API performs flushing and releases any resources held
- flowActivityTable.close();
-}
-if (subApplicationTable != null) {
- subApplicationTable.close();
-}
-if (domainTable != null) {
- domainTable.close();
-}
-if (conn != null) {
- LOG.info("closing the hbase Connection");
- conn.close();
+
+if (isStorageUp) {
+ if (entityTable != null) {
+LOG.info("closing the entity table");
+// The close API performs flushing and releases any resources held
+entityTable.close();
+ }
+ if (appToFlowTable != null) {
+LOG.info("closing the app_flow table");
+// The close API performs flushing and releases any resources held
+appToFlowTable.close();
+ }
+ if (applicationTable != null) {
+LOG.info("closing the application table");
+applicationTable.close();
+ }
+ if (flowRunTable != null) {
+LOG.info("closing the flow run table");
+// The close API performs flushing and releases any resources held
+flowRunTable.close();
+ }
+ if (flowActivityTable != null) {
+LOG.info("closing the flowActivityTable table");
+// The close API performs flushing and releases any resources held
+flowActivityTable.close();
+ }
+ if (subApplicationTable != null) {
+subApplicationTable.close();
+ }
+ if (domainTable != null) {
+domainTable.close();
+ }
+ if (conn != null) {
+LOG.info("closing the hbase Connection");
+conn.close();
+ }
}
storageMonitor.stop();
super.serviceStop();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase is Down. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new f6cc887 YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase
is Down. Contributed by Prabhu Joseph.
f6cc887 is described below
commit f6cc887f3564d57f3df3be47b736299d7bdc2c90
Author: Abhishek Modi
AuthorDate: Mon Sep 9 14:26:00 2019 +0530
YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase is Down.
Contributed by Prabhu Joseph.
---
.../storage/HBaseTimelineWriterImpl.java | 74 --
1 file changed, 42 insertions(+), 32 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineWriterImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineWriterImpl.java
index a398feb..dda004d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineWriterImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineWriterImpl.java
@@ -630,39 +630,49 @@ public class HBaseTimelineWriterImpl extends
AbstractService implements
*/
@Override
protected void serviceStop() throws Exception {
-if (entityTable != null) {
- LOG.info("closing the entity table");
- // The close API performs flushing and releases any resources held
- entityTable.close();
+boolean isStorageUp = true;
+try {
+ storageMonitor.checkStorageIsUp();
+} catch (IOException e) {
+ LOG.warn("Failed to close the timeline tables as Hbase is down", e);
+ isStorageUp = false;
}
-if (appToFlowTable != null) {
- LOG.info("closing the app_flow table");
- // The close API performs flushing and releases any resources held
- appToFlowTable.close();
-}
-if (applicationTable != null) {
- LOG.info("closing the application table");
- applicationTable.close();
-}
-if (flowRunTable != null) {
- LOG.info("closing the flow run table");
- // The close API performs flushing and releases any resources held
- flowRunTable.close();
-}
-if (flowActivityTable != null) {
- LOG.info("closing the flowActivityTable table");
- // The close API performs flushing and releases any resources held
- flowActivityTable.close();
-}
-if (subApplicationTable != null) {
- subApplicationTable.close();
-}
-if (domainTable != null) {
- domainTable.close();
-}
-if (conn != null) {
- LOG.info("closing the hbase Connection");
- conn.close();
+
+if (isStorageUp) {
+ if (entityTable != null) {
+LOG.info("closing the entity table");
+// The close API performs flushing and releases any resources held
+entityTable.close();
+ }
+ if (appToFlowTable != null) {
+LOG.info("closing the app_flow table");
+// The close API performs flushing and releases any resources held
+appToFlowTable.close();
+ }
+ if (applicationTable != null) {
+LOG.info("closing the application table");
+applicationTable.close();
+ }
+ if (flowRunTable != null) {
+LOG.info("closing the flow run table");
+// The close API performs flushing and releases any resources held
+flowRunTable.close();
+ }
+ if (flowActivityTable != null) {
+LOG.info("closing the flowActivityTable table");
+// The close API performs flushing and releases any resources held
+flowActivityTable.close();
+ }
+ if (subApplicationTable != null) {
+subApplicationTable.close();
+ }
+ if (domainTable != null) {
+domainTable.close();
+ }
+ if (conn != null) {
+LOG.info("closing the hbase Connection");
+conn.close();
+ }
}
storageMonitor.stop();
super.serviceStop();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2.1 created (now 865b49a)
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a change to branch branch-3.2.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git. at 865b49a Preparing for 3.2.1 release. This branch includes the following new commits: new 865b49a Preparing for 3.2.1 release. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 01/01: Preparing for 3.2.1 release.
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a commit to branch branch-3.2.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git commit 865b49a54dcc8bf3b4664e4e6376e1eaf8f9598b Author: Rohith Sharma K S AuthorDate: Sat Sep 7 09:08:38 2019 +0530 Preparing for 3.2.1 release. --- hadoop-assemblies/pom.xml | 4 ++-- hadoop-build-tools/pom.xml | 2 +- hadoop-client-modules/hadoop-client-api/pom.xml| 4 ++-- hadoop-client-modules/hadoop-client-check-invariants/pom.xml | 4 ++-- .../hadoop-client-check-test-invariants/pom.xml| 4 ++-- hadoop-client-modules/hadoop-client-integration-tests/pom.xml | 4 ++-- hadoop-client-modules/hadoop-client-minicluster/pom.xml| 4 ++-- hadoop-client-modules/hadoop-client-runtime/pom.xml| 4 ++-- hadoop-client-modules/hadoop-client/pom.xml| 4 ++-- hadoop-client-modules/pom.xml | 2 +- hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml | 4 ++-- hadoop-cloud-storage-project/pom.xml | 4 ++-- hadoop-common-project/hadoop-annotations/pom.xml | 4 ++-- hadoop-common-project/hadoop-auth-examples/pom.xml | 4 ++-- hadoop-common-project/hadoop-auth/pom.xml | 4 ++-- hadoop-common-project/hadoop-common/pom.xml| 4 ++-- hadoop-common-project/hadoop-kms/pom.xml | 4 ++-- hadoop-common-project/hadoop-minikdc/pom.xml | 4 ++-- hadoop-common-project/hadoop-nfs/pom.xml | 4 ++-- hadoop-common-project/pom.xml | 4 ++-- hadoop-dist/pom.xml| 4 ++-- hadoop-hdfs-project/hadoop-hdfs-client/pom.xml | 4 ++-- hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml | 4 ++-- hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml | 4 ++-- hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml| 4 ++-- hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml| 4 ++-- hadoop-hdfs-project/hadoop-hdfs/pom.xml| 4 ++-- hadoop-hdfs-project/pom.xml| 4 ++-- .../hadoop-mapreduce-client-app/pom.xml| 4 ++-- .../hadoop-mapreduce-client-common/pom.xml | 4 ++-- .../hadoop-mapreduce-client-core/pom.xml | 4 ++-- .../hadoop-mapreduce-client-hs-plugins/pom.xml | 4 ++-- .../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml | 4 ++-- .../hadoop-mapreduce-client-jobclient/pom.xml | 4 ++-- .../hadoop-mapreduce-client-nativetask/pom.xml | 4 ++-- .../hadoop-mapreduce-client-shuffle/pom.xml| 4 ++-- .../hadoop-mapreduce-client-uploader/pom.xml | 4 ++-- hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml | 4 ++-- hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml | 4 ++-- hadoop-mapreduce-project/pom.xml | 4 ++-- hadoop-maven-plugins/pom.xml | 2 +- hadoop-minicluster/pom.xml | 4 ++-- hadoop-project-dist/pom.xml| 4 ++-- hadoop-project/pom.xml | 6 +++--- hadoop-tools/hadoop-aliyun/pom.xml | 2 +- hadoop-tools/hadoop-archive-logs/pom.xml | 4 ++-- hadoop-tools/hadoop-archives/pom.xml | 4 ++-- hadoop-tools/hadoop-aws/pom.xml| 4 ++-- hadoop-tools/hadoop-azure-datalake/pom.xml | 2 +- hadoop-tools/hadoop-azure/pom.xml | 2 +- hadoop-tools/hadoop-datajoin/pom.xml | 4 ++-- hadoop-tools/hadoop-distcp/pom.xml | 4 ++-- hadoop-tools/hadoop-extras/pom.xml | 4 ++-- hadoop-tools/hadoop-fs2img/pom.xml | 4 ++-- hadoop-tools/hadoop-gridmix/pom.xml| 4 ++-- hadoop-tools/hadoop-kafka/pom.xml | 4 ++-- hadoop-tools/hadoop-openstack/pom.xml | 4 ++-- hadoop-tools/hadoop-pipes/pom.xml | 4 ++-- hadoop-tools/hadoop-resourceestimator/pom.xml | 2 +- hadoop-tools/hadoop-rumen/pom.xml | 4 ++-- hadoop-tools/hadoop-sls/pom.xml| 4 ++-- hadoop-tools/hadoop-streaming/pom.xml | 4 ++-- hadoop-tools/hadoop-tools-dist/pom.xml | 4 ++-- hadoop
[hadoop] branch branch-3.2 updated: Preparing for 3.2.2-SNAPSHOT development.
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.2 by this push: new 7d5bb2e Preparing for 3.2.2-SNAPSHOT development. 7d5bb2e is described below commit 7d5bb2ebb7be6c87c1aaf4e03eeb6bb36a1be396 Author: Rohith Sharma K S AuthorDate: Sat Sep 7 08:52:08 2019 +0530 Preparing for 3.2.2-SNAPSHOT development. --- hadoop-assemblies/pom.xml | 4 ++-- hadoop-build-tools/pom.xml | 2 +- hadoop-client-modules/hadoop-client-api/pom.xml| 4 ++-- hadoop-client-modules/hadoop-client-check-invariants/pom.xml | 4 ++-- .../hadoop-client-check-test-invariants/pom.xml| 4 ++-- hadoop-client-modules/hadoop-client-integration-tests/pom.xml | 4 ++-- hadoop-client-modules/hadoop-client-minicluster/pom.xml| 4 ++-- hadoop-client-modules/hadoop-client-runtime/pom.xml| 4 ++-- hadoop-client-modules/hadoop-client/pom.xml| 4 ++-- hadoop-client-modules/pom.xml | 2 +- hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml | 4 ++-- hadoop-cloud-storage-project/pom.xml | 4 ++-- hadoop-common-project/hadoop-annotations/pom.xml | 4 ++-- hadoop-common-project/hadoop-auth-examples/pom.xml | 4 ++-- hadoop-common-project/hadoop-auth/pom.xml | 4 ++-- hadoop-common-project/hadoop-common/pom.xml| 4 ++-- hadoop-common-project/hadoop-kms/pom.xml | 4 ++-- hadoop-common-project/hadoop-minikdc/pom.xml | 4 ++-- hadoop-common-project/hadoop-nfs/pom.xml | 4 ++-- hadoop-common-project/pom.xml | 4 ++-- hadoop-dist/pom.xml| 4 ++-- hadoop-hdfs-project/hadoop-hdfs-client/pom.xml | 4 ++-- hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml | 4 ++-- hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml | 4 ++-- hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml| 4 ++-- hadoop-hdfs-project/hadoop-hdfs-rbf/pom.xml| 4 ++-- hadoop-hdfs-project/hadoop-hdfs/pom.xml| 4 ++-- hadoop-hdfs-project/pom.xml| 4 ++-- .../hadoop-mapreduce-client-app/pom.xml| 4 ++-- .../hadoop-mapreduce-client-common/pom.xml | 4 ++-- .../hadoop-mapreduce-client-core/pom.xml | 4 ++-- .../hadoop-mapreduce-client-hs-plugins/pom.xml | 4 ++-- .../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/pom.xml | 4 ++-- .../hadoop-mapreduce-client-jobclient/pom.xml | 4 ++-- .../hadoop-mapreduce-client-nativetask/pom.xml | 4 ++-- .../hadoop-mapreduce-client-shuffle/pom.xml| 4 ++-- .../hadoop-mapreduce-client-uploader/pom.xml | 4 ++-- hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml | 4 ++-- hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml | 4 ++-- hadoop-mapreduce-project/pom.xml | 4 ++-- hadoop-maven-plugins/pom.xml | 2 +- hadoop-minicluster/pom.xml | 4 ++-- hadoop-project-dist/pom.xml| 4 ++-- hadoop-project/pom.xml | 6 +++--- hadoop-tools/hadoop-aliyun/pom.xml | 2 +- hadoop-tools/hadoop-archive-logs/pom.xml | 4 ++-- hadoop-tools/hadoop-archives/pom.xml | 4 ++-- hadoop-tools/hadoop-aws/pom.xml| 4 ++-- hadoop-tools/hadoop-azure-datalake/pom.xml | 2 +- hadoop-tools/hadoop-azure/pom.xml | 2 +- hadoop-tools/hadoop-datajoin/pom.xml | 4 ++-- hadoop-tools/hadoop-distcp/pom.xml | 4 ++-- hadoop-tools/hadoop-extras/pom.xml | 4 ++-- hadoop-tools/hadoop-fs2img/pom.xml | 4 ++-- hadoop-tools/hadoop-gridmix/pom.xml| 4 ++-- hadoop-tools/hadoop-kafka/pom.xml | 4 ++-- hadoop-tools/hadoop-openstack/pom.xml | 4 ++-- hadoop-tools/hadoop-pipes/pom.xml | 4 ++-- hadoop-tools/hadoop-resourceestimator/pom.xml | 2 +- hadoop-tools/hadoop-rumen/pom.xml | 4 ++-- hadoop-tools/hadoop-sls/pom.xml
[hadoop] 03/05: YARN-8499 ATSv2 Generalize TimelineStorageMonitor. Contributed by Prabhu Joseph
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 02779cdc3a1238991179373395b092cd6321765b
Author: Eric Yang
AuthorDate: Fri Jun 14 18:59:14 2019 -0400
YARN-8499 ATSv2 Generalize TimelineStorageMonitor.
Contributed by Prabhu Joseph
---
.../storage/TestTimelineReaderHBaseDown.java | 4 +-
.../storage/HBaseStorageMonitor.java | 90 +
.../storage/HBaseTimelineReaderImpl.java | 92 ++
.../storage/TimelineStorageMonitor.java| 106 +
4 files changed, 206 insertions(+), 86 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineReaderHBaseDown.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineReaderHBaseDown.java
index 786f529..e738d39 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineReaderHBaseDown.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineReaderHBaseDown.java
@@ -34,8 +34,8 @@ import java.util.Set;
import java.util.concurrent.TimeoutException;
import static
org.apache.hadoop.yarn.conf.YarnConfiguration.TIMELINE_SERVICE_READER_STORAGE_MONITOR_INTERVAL_MS;
-import static
org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.DATA_TO_RETRIEVE;
-import static
org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.MONITOR_FILTERS;
+import static
org.apache.hadoop.yarn.server.timelineservice.storage.HBaseStorageMonitor.DATA_TO_RETRIEVE;
+import static
org.apache.hadoop.yarn.server.timelineservice.storage.HBaseStorageMonitor.MONITOR_FILTERS;
public class TestTimelineReaderHBaseDown {
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseStorageMonitor.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseStorageMonitor.java
new file mode 100644
index 000..c433aa6
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseStorageMonitor.java
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.timelineservice.storage;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntityType;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineDataToRetrieve;
+import
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineEntityFilters;
+import
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderContext;
+import
org.apache.hadoop.yarn.server.timelineservice.storage.common.HBaseTimelineStorageUtils;
+import
org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader;
+import
org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReaderFactory;
+
+/**
+ * HBase based implementation for {@link TimelineStorageMonitor}.
+ */
+public class HBaseStorageMonitor extends TimelineStorageMonitor {
+
+ pr
[hadoop] 02/05: YARN-7537. Add ability to load hbase config from distributed file system. Contributed by Prabhu Joseph
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 6110af2d1df75d37a2d4e4095e132df23c4ed429
Author: Eric Yang
AuthorDate: Tue Jun 4 19:26:06 2019 -0400
YARN-7537. Add ability to load hbase config from distributed file system.
Contributed by Prabhu Joseph
---
.../pom.xml| 13
.../storage/common/HBaseTimelineStorageUtils.java | 32 ++
.../common/TestHBaseTimelineStorageUtils.java | 74 ++
3 files changed, 106 insertions(+), 13 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml
index ff7fb6e..74af322 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml
@@ -76,6 +76,19 @@
org.apache.hadoop
+ hadoop-hdfs
+ test
+
+
+
+ org.apache.hadoop
+ hadoop-hdfs
+ test-jar
+ test
+
+
+
+ org.apache.hadoop
hadoop-yarn-api
provided
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/HBaseTimelineStorageUtils.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/HBaseTimelineStor
[...]
index f4cd6fb..93feb82 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/HBaseTimelineStorageUtils.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/HBaseTimelineStorageUtils.java
@@ -17,11 +17,13 @@
package org.apache.hadoop.yarn.server.timelineservice.storage.common;
-import java.net.MalformedURLException;
-import java.net.URL;
+import java.io.IOException;
import java.util.Arrays;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.client.Query;
@@ -40,7 +42,6 @@ public final class HBaseTimelineStorageUtils {
private HBaseTimelineStorageUtils() {
}
-
/**
* @param conf YARN configuration. Used to see if there is an explicit config
* pointing to the HBase config file to read. It should not be null
@@ -48,28 +49,33 @@ public final class HBaseTimelineStorageUtils {
* @return a configuration with the HBase configuration from the classpath,
* optionally overwritten by the timeline service configuration URL
if
* specified.
- * @throws MalformedURLException if a timeline service HBase configuration
URL
- * is specified but is a malformed URL.
+ * @throws IOException if a timeline service HBase configuration URL
+ * is specified but unable to read it.
*/
public static Configuration getTimelineServiceHBaseConf(Configuration conf)
- throws MalformedURLException {
+ throws IOException {
if (conf == null) {
throw new NullPointerException();
}
Configuration hbaseConf;
-String timelineServiceHBaseConfFileURL =
+String timelineServiceHBaseConfFilePath =
conf.get(YarnConfiguration.TIMELINE_SERVICE_HBASE_CONFIGURATION_FILE);
-if (timelineServiceHBaseConfFileURL != null
-&& timelineServiceHBaseConfFileURL.length() > 0) {
+
+if (timelineServiceHBaseConfFilePath != null
+ && timelineServiceHBaseConfFilePath.length() > 0) {
LOG.info("Using hbase configuration at " +
- timelineServiceHBaseConfFileURL);
+ timelineServiceHBaseConfFilePath);
// create a clone so that we don't mess with out input one
hbaseConf = new Configuration(conf);
Configuration plainHBaseConf = new Configuration(false);
- URL hbaseSiteXML = new URL
[hadoop] 04/05: YARN-9374. Improve Timeline service resilience when HBase is unavailable. Contributed by Prabhu Joseph and Szilard Nemeth
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit b87a727ff439da246cde9338fa2cd6495ec09b37
Author: Eric Yang
AuthorDate: Mon Jun 24 12:19:14 2019 -0400
YARN-9374. Improve Timeline service resilience when HBase is unavailable.
Contributed by Prabhu Joseph and Szilard Nemeth
---
.../storage/TestTimelineReaderHBaseDown.java | 18 +++-
.../storage/TestTimelineWriterHBaseDown.java | 117 +
.../storage/HBaseTimelineReaderImpl.java | 13 +--
.../storage/HBaseTimelineWriterImpl.java | 19 +++-
.../storage/TimelineStorageMonitor.java| 4 -
5 files changed, 158 insertions(+), 13 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineReaderHBaseDown.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineReaderHBaseDown.java
index e738d39..1148b80 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineReaderHBaseDown.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineReaderHBaseDown.java
@@ -150,7 +150,14 @@ public class TestTimelineReaderHBaseDown {
waitForHBaseDown(htr);
util.startMiniHBaseCluster(1, 1);
- GenericTestUtils.waitFor(() -> !htr.isHBaseDown(), 1000, 15);
+ GenericTestUtils.waitFor(() -> {
+try {
+ htr.getTimelineStorageMonitor().checkStorageIsUp();
+ return true;
+} catch (IOException e) {
+ return false;
+}
+ }, 1000, 15);
} finally {
util.shutdownMiniCluster();
}
@@ -158,8 +165,15 @@ public class TestTimelineReaderHBaseDown {
private static void waitForHBaseDown(HBaseTimelineReaderImpl htr) throws
TimeoutException, InterruptedException {
-GenericTestUtils.waitFor(() -> htr.isHBaseDown(), 1000, 15);
try {
+ GenericTestUtils.waitFor(() -> {
+try {
+ htr.getTimelineStorageMonitor().checkStorageIsUp();
+ return false;
+} catch (IOException e) {
+ return true;
+}
+ }, 1000, 15);
checkQuery(htr);
Assert.fail("Query should fail when HBase is down");
} catch (IOException e) {
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineWriterHBaseDown.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineWriterHBaseDown.java
new file mode 100644
index 000..cb89ba4
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineWriterHBaseDown.java
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.timelineservice.storage;
+
+import java.io.IOException;
+
+import org.junit.Test;
+import org.junit.Assert;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.test.GenericTestUtils;
+import
org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorContext;
+import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities;
+import org.apache.hadoop.yarn.api.records.timelines
[hadoop] 05/05: YARN-8567. Fetching yarn logs fails for long running application if it is not present in timeline store. Contributed by Tarun Parimi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 4d9c5300e2341e954e3c01bd364388df265276cc
Author: Rohith Sharma K S
AuthorDate: Fri Jan 4 14:01:24 2019 +0530
YARN-8567. Fetching yarn logs fails for long running application if it is
not present in timeline store. Contributed by Tarun Parimi.
---
.../yarn/client/api/impl/YarnClientImpl.java | 2 +-
.../yarn/client/api/impl/TestYarnClient.java | 50 ++
2 files changed, 51 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
index 227f7ed..76cd196 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
@@ -870,7 +870,7 @@ public class YarnClientImpl extends YarnClient {
try {
containersListFromAHS =
getContainerReportFromHistory(applicationAttemptId);
-} catch (IOException e) {
+} catch (IOException | YarnException e) {
if (appNotFoundInRM) {
throw e;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
index 17e43ca..7ad7e3a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
@@ -531,6 +531,44 @@ public class TestYarnClient extends
ParameterizedSchedulerTestBase {
}
@Test(timeout = 1)
+ public void testGetContainersOnAHSFail() throws YarnException, IOException {
+Configuration conf = getConf();
+conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+true);
+
+final YarnClient client = new MockYarnClient() {
+ @Override
+ public List getContainers(
+ ApplicationAttemptId appAttemptId) throws YarnException,
+ IOException {
+return getContainersOnAHSFail(appAttemptId);
+ }
+};
+
+client.init(conf);
+client.start();
+
+ApplicationId applicationId = ApplicationId.newInstance(1234, 5);
+ApplicationAttemptId appAttemptId = ApplicationAttemptId.newInstance(
+applicationId, 1);
+List reports = client.getContainers(appAttemptId);
+Assert.assertNotNull(reports);
+Assert.assertTrue(reports.size() == 2);
+Assert.assertEquals(reports.get(0).getContainerId(),
+(ContainerId.newContainerId(appAttemptId, 1)));
+Assert.assertEquals(reports.get(1).getContainerId(),
+(ContainerId.newContainerId(appAttemptId, 2)));
+
+//Only 2 running containers from RM are present when AHS throws exception
+Assert.assertEquals(ContainerState.RUNNING,
+(reports.get(0).getContainerState()));
+Assert.assertEquals(ContainerState.RUNNING,
+(reports.get(1).getContainerState()));
+client.stop();
+ }
+
+
+ @Test(timeout = 1)
public void testGetContainerReport() throws YarnException, IOException {
Configuration conf = getConf();
conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
@@ -914,6 +952,18 @@ public class TestYarnClient extends
ParameterizedSchedulerTestBase {
return super.getContainers(appAttemptId);
}
+protected List
+getContainersOnAHSFail(ApplicationAttemptId appAttemptId)
+ throws YarnException, IOException {
+ when(mockContainersResponse.getContainerList()).thenReturn(
+ getContainersReport(appAttemptId));
+ when(historyClient.getContainers(any(ApplicationAttemptId.class)))
+ .thenThrow(new ApplicationNotFoundException(
+ appAttemptId.getApplicationId() +
+ " does not exist in the timeline store"));
+ return super.getContainers(appAttemptId);
+}
+
private List getContainersFromAHS(
ApplicationAttemptId appAttemptId) {
return containersFromAHS.get(appAttemptId);
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated (6acc1a2 -> 4d9c530)
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a change to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 6acc1a2 YARN-9382 Publish container killed, paused and resumed events to ATSv2. Contributed by Abhishesk Modi. new bcacb57 YARN-9335 [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls. Contributed by Abhishesk Modi. new 6110af2 YARN-7537. Add ability to load hbase config from distributed file system. Contributed by Prabhu Joseph new 02779cd YARN-8499 ATSv2 Generalize TimelineStorageMonitor. Contributed by Prabhu Joseph new b87a727 YARN-9374. Improve Timeline service resilience when HBase is unavailable. Contributed by Prabhu Joseph and Szilard Nemeth new 4d9c530 YARN-8567. Fetching yarn logs fails for long running application if it is not present in timeline store. Contributed by Tarun Parimi. The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/hadoop/yarn/conf/YarnConfiguration.java | 9 ++ .../yarn/client/api/impl/YarnClientImpl.java | 2 +- .../yarn/client/api/impl/TestYarnClient.java | 50 + .../src/main/resources/yarn-default.xml| 7 ++ .../storage/TestTimelineReaderHBaseDown.java | 22 +++- .../storage/TestTimelineWriterHBaseDown.java | 117 + .../pom.xml| 13 +++ .../storage/HBaseStorageMonitor.java | 90 .../storage/HBaseTimelineReaderImpl.java | 101 +++--- .../storage/HBaseTimelineWriterImpl.java | 19 +++- .../storage/common/HBaseTimelineStorageUtils.java | 32 +++--- .../common/TestHBaseTimelineStorageUtils.java | 74 + .../collector/TimelineCollector.java | 24 - .../storage/TimelineStorageMonitor.java| 102 ++ .../collector/TestTimelineCollector.java | 51 - 15 files changed, 602 insertions(+), 111 deletions(-) create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestTimelineWriterHBaseDown.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseStorageMonitor.java create mode 100644 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/TimelineStorageMonitor.java - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 01/05: YARN-9335 [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls. Contributed by Abhishesk Modi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit bcacb5711425f938c87dc985afdc5ec2464d58f2
Author: Vrushali C
AuthorDate: Fri Apr 5 12:06:51 2019 -0700
YARN-9335 [atsv2] Restrict the number of elements held in timeline
collector when backend is unreachable for async calls. Contributed by Abhishesk
Modi.
---
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 9
.../src/main/resources/yarn-default.xml| 7 +++
.../collector/TimelineCollector.java | 24 +-
.../collector/TestTimelineCollector.java | 51 --
4 files changed, 87 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 90d4298..c69d857 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -2706,6 +2706,15 @@ public class YarnConfiguration extends Configuration {
public static final int
DEFAULT_TIMELINE_SERVICE_WRITER_FLUSH_INTERVAL_SECONDS = 60;
+ /** The setting that controls the capacity of the queue for async writes
+ * to timeline collector.
+ */
+ public static final String TIMELINE_SERVICE_WRITER_ASYNC_QUEUE_CAPACITY =
+ TIMELINE_SERVICE_PREFIX + "writer.async.queue.capacity";
+
+ public static final int
+ DEFAULT_TIMELINE_SERVICE_WRITER_ASYNC_QUEUE_CAPACITY = 100;
+
/**
* The name for setting that controls how long the final value of
* a metric of a completed app is retained before merging
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index 72125b1..3b9cf5d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -2568,6 +2568,13 @@
+The setting that decides the capacity of the queue to hold
+asynchronous timeline entities.
+yarn.timeline-service.writer.async.queue.capacity
+100
+
+
+
Time period till which the application collector will be alive
in NM, after the application master container finishes.
yarn.timeline-service.app-collector.linger-period.ms
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollector.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollector.java
index e9eeb43..c0004e5 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollector.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollector.java
@@ -23,8 +23,11 @@ import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
+import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
import org.apache.hadoop.classification.InterfaceAudience.Private;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
@@ -37,6 +40,7 @@ import
org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineMetric;
import
org.apache.hadoop.yarn.api.records.timelineservice.TimelineWriteResponse;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.server.timelineservice.storage.TimelineWriter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -61,6 +65,7 @@ public abstract class TimelineCollector extends
CompositeService {
= new ConcurrentHashMap<>();
private static Set entityTypesSkipAggregation
= new HashSet<>();
+ private ThreadPoolExecutor pool;
private volatile boolean readyToAggregate = false;
@@ -73,6 +78,14 @@ public abstract class TimelineCollector extends
CompositeService {
@Override
protected void serviceInit(Conf
[hadoop] 05/07: YARN-9418. ATSV2 /apps//entities/YARN_CONTAINER rest api does not show metrics. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 998aa3de2cee8ab40b2297e52a79978c52fbb933
Author: Giovanni Matteo Fumarola
AuthorDate: Mon Apr 1 11:06:51 2019 -0700
YARN-9418. ATSV2 /apps//entities/YARN_CONTAINER rest api does not show
metrics. Contributed by Prabhu Joseph.
---
.../timelineservice/NMTimelinePublisher.java | 8 ++--
.../timelineservice/TestNMTimelinePublisher.java | 20 +---
.../metrics/TimelineServiceV2Publisher.java | 2 ++
.../metrics/TestSystemMetricsPublisherForV2.java | 3 ++-
4 files changed, 19 insertions(+), 14 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
index e9bd965..18e3d86 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
@@ -186,8 +186,6 @@ public class NMTimelinePublisher extends CompositeService {
Math.round(cpuUsagePercentPerCore));
entity.addMetric(cpuMetric);
}
-entity.setIdPrefix(TimelineServiceHelper.
-invertLong(container.getContainerStartTime()));
ApplicationId appId = container.getContainerId().
getApplicationAttemptId().getApplicationId();
try {
@@ -253,7 +251,6 @@ public class NMTimelinePublisher extends CompositeService {
long containerStartTime = container.getContainerStartTime();
entity.addEvent(tEvent);
entity.setCreatedTime(containerStartTime);
- entity.setIdPrefix(TimelineServiceHelper.invertLong(containerStartTime));
dispatcher.getEventHandler().handle(new TimelinePublishEvent(entity,
containerId.getApplicationAttemptId().getApplicationId()));
}
@@ -281,7 +278,6 @@ public class NMTimelinePublisher extends CompositeService {
tEvent.setId(ContainerMetricsConstants.FINISHED_EVENT_TYPE);
tEvent.setTimestamp(containerFinishTime);
entity.addEvent(tEvent);
- entity.setIdPrefix(TimelineServiceHelper.invertLong(containerStartTime));
dispatcher.getEventHandler().handle(new TimelinePublishEvent(entity,
containerId.getApplicationAttemptId().getApplicationId()));
@@ -299,8 +295,6 @@ public class NMTimelinePublisher extends CompositeService {
tEvent.setId(eventType);
tEvent.setTimestamp(event.getTimestamp());
entity.addEvent(tEvent);
- entity.setIdPrefix(TimelineServiceHelper.
- invertLong(container.getContainerStartTime()));
ApplicationId appId = container.getContainerId().
getApplicationAttemptId().getApplicationId();
@@ -336,6 +330,8 @@ public class NMTimelinePublisher extends CompositeService {
ContainerId containerId) {
ContainerEntity entity = new ContainerEntity();
entity.setId(containerId.toString());
+entity.setIdPrefix(TimelineServiceHelper.invertLong(
+containerId.getContainerId()));
Identifier parentIdentifier = new Identifier();
parentIdentifier
.setType(TimelineEntityType.YARN_APPLICATION_ATTEMPT.name());
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java
index cf9ede0..ae51f85 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java
@@ -46,6 +46,7 @@ import org.apache.hadoop.yarn.server.nodemanager.Context;
import
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationContainerFinishedEvent;
import
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container;
import org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree;
+import org.apache.hadoop.yarn.util.TimelineServiceHelper;
import org.junit.Assert;
import
[hadoop] branch branch-3.2 updated (6f97640 -> 6acc1a2)
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a change to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 6f97640 YARN-8995. Log events info in AsyncDispatcher when event queue size cumulatively reaches a certain number every time. Contributed by zhuqi. new 66e1599 YARN-3879 [Storage implementation] Create HDFS backing storage implementation for ATS reads. Contributed by Abhishek Modi. new d451ff7 YARN-3841 [atsv2 Storage implementation] Adding retry semantics to HDFS backing storage. Contributed by Abhishek Modi. new 0ccc5a2 YARN-9387. Update document for ATS HBase Custom tablenames (-entityTableName). Contributed by Prabhu Joseph. new 8de93fc YARN-9389. FlowActivity and FlowRun table prefix is wrong. Contributed by Prabhu Joseph. new 998aa3d YARN-9418. ATSV2 /apps//entities/YARN_CONTAINER rest api does not show metrics. Contributed by Prabhu Joseph. new f52a88f YARN-9303 Username splits won't help timelineservice.app_flow table. Contributed by Prabhu Joseph. new 6acc1a2 YARN-9382 Publish container killed, paused and resumed events to ATSv2. Contributed by Abhishesk Modi. The 7 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../server/metrics/ContainerMetricsConstants.java | 9 + .../timelineservice/NMTimelinePublisher.java | 110 ++- .../timelineservice/TestNMTimelinePublisher.java | 157 ++- .../metrics/TimelineServiceV2Publisher.java| 2 + .../metrics/TestSystemMetricsPublisherForV2.java | 3 +- .../storage/apptoflow/AppToFlowTableRW.java| 8 +- .../storage/flow/FlowActivityTableRW.java | 2 +- .../storage/flow/FlowRunTableRW.java | 2 +- .../storage/FileSystemTimelineReaderImpl.java | 108 +- .../storage/FileSystemTimelineWriterImpl.java | 217 - .../storage/TestFileSystemTimelineWriterImpl.java | 51 +++-- .../src/site/markdown/TimelineServiceV2.md | 19 +- 12 files changed, 546 insertions(+), 142 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 02/07: YARN-3841 [atsv2 Storage implementation] Adding retry semantics to HDFS backing storage. Contributed by Abhishek Modi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit d451ff753423915b6103f980d0f397325e4cc38e
Author: Vrushali C
AuthorDate: Wed Feb 27 14:55:35 2019 -0800
YARN-3841 [atsv2 Storage implementation] Adding retry semantics to HDFS
backing storage. Contributed by Abhishek Modi.
---
.../storage/FileSystemTimelineWriterImpl.java | 217 -
.../storage/TestFileSystemTimelineWriterImpl.java | 51 +++--
2 files changed, 207 insertions(+), 61 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FileSystemTimelineWriterImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FileSystemTimelineWriterImpl.java
index ac0902f..c284f8f 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FileSystemTimelineWriterImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FileSystemTimelineWriterImpl.java
@@ -18,16 +18,16 @@
package org.apache.hadoop.yarn.server.timelineservice.storage;
-import java.io.BufferedWriter;
import java.io.File;
-import java.io.FileOutputStream;
import java.io.IOException;
-import java.io.OutputStreamWriter;
-import java.io.PrintWriter;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.service.AbstractService;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineDomain;
@@ -35,14 +35,17 @@ import
org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity;
import
org.apache.hadoop.yarn.api.records.timelineservice.TimelineWriteResponse;
import
org.apache.hadoop.yarn.api.records.timelineservice.TimelineWriteResponse.TimelineWriteError;
+import org.apache.hadoop.yarn.client.api.impl.FileSystemTimelineWriter;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import
org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorContext;
import org.apache.hadoop.yarn.util.timeline.TimelineUtils;
import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
- * This implements a local file based backend for storing application timeline
+ * This implements a FileSystem based backend for storing application timeline
* information. This implementation may not provide a complete implementation
of
* all the necessary features. This implementation is provided solely for basic
* testing purposes, and should not be used in a non-test situation.
@@ -52,20 +55,36 @@ import com.google.common.annotations.VisibleForTesting;
public class FileSystemTimelineWriterImpl extends AbstractService
implements TimelineWriter {
- private String outputRoot;
-
/** Config param for timeline service storage tmp root for FILE YARN-3264. */
- public static final String TIMELINE_SERVICE_STORAGE_DIR_ROOT
- = YarnConfiguration.TIMELINE_SERVICE_PREFIX + "fs-writer.root-dir";
+ public static final String TIMELINE_SERVICE_STORAGE_DIR_ROOT =
+ YarnConfiguration.TIMELINE_SERVICE_PREFIX + "fs-writer.root-dir";
+
+ public static final String TIMELINE_FS_WRITER_NUM_RETRIES =
+ YarnConfiguration.TIMELINE_SERVICE_PREFIX + "fs-writer.num-retries";
+ public static final int DEFAULT_TIMELINE_FS_WRITER_NUM_RETRIES = 0;
+
+ public static final String TIMELINE_FS_WRITER_RETRY_INTERVAL_MS =
+ YarnConfiguration.TIMELINE_SERVICE_PREFIX +
+ "fs-writer.retry-interval-ms";
+ public static final long DEFAULT_TIMELINE_FS_WRITER_RETRY_INTERVAL_MS =
1000L;
public static final String ENTITIES_DIR = "entities";
/** Default extension for output files. */
public static final String TIMELINE_SERVICE_STORAGE_EXTENSION = ".thist";
+ private FileSystem fs;
+ private Path rootPath;
+ private int fsNumRetries;
+ private long fsRetryInterval;
+ private Path entitiesPath;
+
/** default value for storage location on local disk. */
private static final String STORAGE_DIR_ROOT = "timeline_service_data";
+ private static final Logger L
[hadoop] 04/07: YARN-9389. FlowActivity and FlowRun table prefix is wrong. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 8de93fca3c3748c6cb5548a7522aa17c699a0c60
Author: Rohith Sharma K S
AuthorDate: Wed Mar 20 21:18:19 2019 +0530
YARN-9389. FlowActivity and FlowRun table prefix is wrong. Contributed by
Prabhu Joseph.
---
.../yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java | 2 +-
.../hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java
index 5b9fe13..ead6a79 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java
@@ -36,7 +36,7 @@ import org.slf4j.LoggerFactory;
public class FlowActivityTableRW extends BaseTableRW {
/** flow activity table prefix. */
private static final String PREFIX =
- YarnConfiguration.TIMELINE_SERVICE_PREFIX + ".flowactivity";
+ YarnConfiguration.TIMELINE_SERVICE_PREFIX + "flowactivity";
/** config param name that specifies the flowactivity table name. */
public static final String TABLE_NAME_CONF_NAME = PREFIX + ".table.name";
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java
index 61c0734..b3e81a1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java
@@ -38,7 +38,7 @@ import org.apache.hadoop.hbase.Coprocessor;
public class FlowRunTableRW extends BaseTableRW {
/** entity prefix. */
private static final String PREFIX =
- YarnConfiguration.TIMELINE_SERVICE_PREFIX + ".flowrun";
+ YarnConfiguration.TIMELINE_SERVICE_PREFIX + "flowrun";
/** config param name that specifies the flowrun table name. */
public static final String TABLE_NAME_CONF_NAME = PREFIX + ".table.name";
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 07/07: YARN-9382 Publish container killed, paused and resumed events to ATSv2. Contributed by Abhishesk Modi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 6acc1a2bd0acca4dbebc0a7d143ac07c6b3c6d94
Author: Vrushali C
AuthorDate: Fri Apr 5 12:02:43 2019 -0700
YARN-9382 Publish container killed, paused and resumed events to ATSv2.
Contributed by Abhishesk Modi.
---
.../server/metrics/ContainerMetricsConstants.java | 9 ++
.../timelineservice/NMTimelinePublisher.java | 102 ++-
.../timelineservice/TestNMTimelinePublisher.java | 137 +
3 files changed, 247 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java
index 9cf2b0a..163171f 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ContainerMetricsConstants.java
@@ -35,6 +35,15 @@ public class ContainerMetricsConstants {
"YARN_RM_CONTAINER_CREATED";
// Event of this type will be emitted by NM.
+ public static final String PAUSED_EVENT_TYPE = "YARN_CONTAINER_PAUSED";
+
+ // Event of this type will be emitted by NM.
+ public static final String RESUMED_EVENT_TYPE = "YARN_CONTAINER_RESUMED";
+
+ // Event of this type will be emitted by NM.
+ public static final String KILLED_EVENT_TYPE = "YARN_CONTAINER_KILLED";
+
+ // Event of this type will be emitted by NM.
public static final String FINISHED_EVENT_TYPE = "YARN_CONTAINER_FINISHED";
// Event of this type will be emitted by RM.
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
index 18e3d86..eb77b7c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
@@ -25,6 +25,9 @@ import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerKillEvent;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerPauseEvent;
+import
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerResumeEvent;
import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -257,6 +260,95 @@ public class NMTimelinePublisher extends CompositeService {
}
@SuppressWarnings("unchecked")
+ private void publishContainerResumedEvent(
+ ContainerEvent event) {
+if (publishNMContainerEvents) {
+ ContainerResumeEvent resumeEvent = (ContainerResumeEvent) event;
+ ContainerId containerId = resumeEvent.getContainerID();
+ ContainerEntity entity = createContainerEntity(containerId);
+
+ Map entityInfo = new HashMap();
+ entityInfo.put(ContainerMetricsConstants.DIAGNOSTICS_INFO,
+ resumeEvent.getDiagnostic());
+ entity.setInfo(entityInfo);
+
+ Container container = context.getContainers().get(containerId);
+ if (container != null) {
+TimelineEvent tEvent = new TimelineEvent();
+tEvent.setId(ContainerMetricsConstants.RESUMED_EVENT_TYPE);
+tEvent.setTimestamp(event.getTimestamp());
+
+long containerStartTime = container.getContainerStartTime();
+entity.addEvent(tEvent);
+entity
+.setIdPrefix(TimelineServiceHelper.invertLong(containerStartTime));
+dispatcher.getEventHandler().handle(new TimelinePublishEvent(entity,
+containerId.getApplicationAttemptId().getApplicationId()));
+ }
+}
+ }
+
+ @SuppressWarnings("unchecked")
+ private void publishContainerPausedEvent(
+ ContainerEvent event) {
+if (publishNMContainerEvents) {
+ ContainerPauseEvent pauseEvent = (ContainerPauseEvent) event;
+ ContainerId containerId = pauseEvent.getContainerID();
+ ContainerEntity entity = createContainerEntity(containerId);
+
+ Map entityInfo = new HashM
[hadoop] 03/07: YARN-9387. Update document for ATS HBase Custom tablenames (-entityTableName). Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 0ccc5a26950cbe6b676d57c401f4c0887da4eac6
Author: Rohith Sharma K S
AuthorDate: Wed Mar 20 21:20:50 2019 +0530
YARN-9387. Update document for ATS HBase Custom tablenames
(-entityTableName). Contributed by Prabhu Joseph.
---
.../src/site/markdown/TimelineServiceV2.md| 19 +++
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
index 457158d..a6c922e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
@@ -259,10 +259,21 @@ The `TimelineSchemaCreator` tool supports a few options
that may come handy espe
are testing. For example, you can use `-skipExistingTable` (`-s` for short) to
skip existing tables
and continue to create other tables rather than failing the schema creation.
By default, the tables
will have a schema prefix of "prod.". When no option or '-help' ('-h' for
short) is provided, the
-command usage is printed.
-and continue to create other tables rather than failing the schema creation.
When no option or '-help'
-('-h' for short) is provided, the command usage is printed. By default, the
tables
-will have a schema prefix of "prod."
+command usage is printed. The options (-entityTableName, -appToflowTableName,
-applicationTableName,
+-subApplicationTableName) will help to override the default table names. On
using custom table names,
+The below corresponding configs with custom table name has to be set in
hbase-site.xml configured
+at yarn.timeline-service.hbase.configuration.file.
+
+```
+yarn.timeline-service.app-flow.table.name
+yarn.timeline-service.entity.table.name
+yarn.timeline-service.application.table.name
+yarn.timeline-service.subapplication.table.name
+yarn.timeline-service.flowactivity.table.name
+yarn.timeline-service.flowrun.table.name
+yarn.timeline-service.domain.table.name
+
+```
Enabling Timeline Service v.2
Following are the basic configurations to start Timeline service v.2:
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 06/07: YARN-9303 Username splits won't help timelineservice.app_flow table. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit f52a88fdc89889fbed05fe573bb0fb6e2c610ee5
Author: Vrushali C
AuthorDate: Wed Apr 3 22:53:05 2019 -0700
YARN-9303 Username splits won't help timelineservice.app_flow table.
Contributed by Prabhu Joseph.
---
.../timelineservice/storage/apptoflow/AppToFlowTableRW.java | 8 +---
1 file changed, 1 insertion(+), 7 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/apptoflow/AppToFlowTableRW.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/apptoflow/AppToFlowTableRW.java
index 6460203..05c4e57 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/apptoflow/AppToFlowTableRW.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/apptoflow/AppToFlowTableRW.java
@@ -79,13 +79,7 @@ public class AppToFlowTableRW extends
BaseTableRW {
mappCF.setBloomFilterType(BloomType.ROWCOL);
appToFlowTableDescp.addFamily(mappCF);
-appToFlowTableDescp
-.setRegionSplitPolicyClassName(
-"org.apache.hadoop.hbase.regionserver.KeyPrefixRegionSplitPolicy");
-appToFlowTableDescp.setValue("KeyPrefixRegionSplitPolicy.prefix_length",
-TimelineHBaseSchemaConstants.USERNAME_SPLIT_KEY_PREFIX_LENGTH);
-admin.createTable(appToFlowTableDescp,
-TimelineHBaseSchemaConstants.getUsernameSplits());
+admin.createTable(appToFlowTableDescp);
LOG.info("Status of table creation for " + table.getNameAsString() + "="
+ admin.tableExists(table));
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 01/07: YARN-3879 [Storage implementation] Create HDFS backing storage implementation for ATS reads. Contributed by Abhishek Modi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 66e1599761ed1fa80d564f06b4b90815f64fe93b
Author: Vrushali C
AuthorDate: Thu Oct 11 21:13:52 2018 -0700
YARN-3879 [Storage implementation] Create HDFS backing storage
implementation for ATS reads. Contributed by Abhishek Modi.
---
.../storage/FileSystemTimelineReaderImpl.java | 108 +++--
1 file changed, 55 insertions(+), 53 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FileSystemTimelineReaderImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FileSystemTimelineReaderImpl.java
index cea70af..d1b7e86 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FileSystemTimelineReaderImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FileSystemTimelineReaderImpl.java
@@ -20,7 +20,6 @@ package org.apache.hadoop.yarn.server.timelineservice.storage;
import java.io.BufferedReader;
import java.io.File;
-import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;
@@ -41,6 +40,11 @@ import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVRecord;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.LocatedFileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
import org.apache.hadoop.service.AbstractService;
import org.apache.hadoop.yarn.api.records.timeline.TimelineHealth;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity;
@@ -69,7 +73,9 @@ public class FileSystemTimelineReaderImpl extends
AbstractService
private static final Logger LOG =
LoggerFactory.getLogger(FileSystemTimelineReaderImpl.class);
- private String rootPath;
+ private FileSystem fs;
+ private Path rootPath;
+ private Path entitiesPath;
private static final String ENTITIES_DIR = "entities";
/** Default extension for output files. */
@@ -95,7 +101,7 @@ public class FileSystemTimelineReaderImpl extends
AbstractService
@VisibleForTesting
String getRootPath() {
-return rootPath;
+return rootPath.toString();
}
private static ObjectMapper mapper;
@@ -163,12 +169,12 @@ public class FileSystemTimelineReaderImpl extends
AbstractService
if (clusterId == null || appId == null) {
throw new IOException("Unable to get flow info");
}
-String appFlowMappingFile = rootPath + File.separator + ENTITIES_DIR +
-File.separator + clusterId + File.separator + APP_FLOW_MAPPING_FILE;
+Path clusterIdPath = new Path(entitiesPath, clusterId);
+Path appFlowMappingFilePath = new Path(clusterIdPath,
+APP_FLOW_MAPPING_FILE);
try (BufferedReader reader =
new BufferedReader(new InputStreamReader(
- new FileInputStream(
- appFlowMappingFile), Charset.forName("UTF-8")));
+ fs.open(appFlowMappingFilePath), Charset.forName("UTF-8")));
CSVParser parser = new CSVParser(reader, csvFormat)) {
for (CSVRecord record : parser.getRecords()) {
if (record.size() < 4) {
@@ -267,7 +273,7 @@ public class FileSystemTimelineReaderImpl extends
AbstractService
return entity;
}
- private Set getEntities(File dir, String entityType,
+ private Set getEntities(Path dir, String entityType,
TimelineEntityFilters filters, TimelineDataToRetrieve dataToRetrieve)
throws IOException {
// First sort the selected entities based on created/start time.
@@ -281,15 +287,18 @@ public class FileSystemTimelineReaderImpl extends
AbstractService
}
);
if (dir != null) {
- File[] files = dir.listFiles();
- if (files != null) {
-for (File entityFile : files) {
+ RemoteIterator fileStatuses = fs.listFiles(dir,
+ false);
+ if (fileStatuses != null) {
+while (fileStatuses.hasNext()) {
+ LocatedFileStatus locatedFileStatus = fileStatuses.next();
+ Path entityFile = locatedFileStatus.getPath();
if (!entityFile.getName()
.contains(TIMELINE_SERVICE_STORAGE_EXTENSION)) {
continue;
}
try (BufferedReader reader = new BufferedReade
[hadoop] 04/06: YARN-6149. Allow port range to be specified while starting NM Timeline collector manager. Contributed by Abhishek Modi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 5345508fa33782bcfeb0cbb378ec210fa2a45708
Author: Rohith Sharma K S
AuthorDate: Fri Jan 4 14:07:02 2019 +0530
YARN-6149. Allow port range to be specified while starting NM Timeline
collector manager. Contributed by Abhishek Modi.
---
.../org/apache/hadoop/yarn/conf/YarnConfiguration.java| 4
.../collector/NodeTimelineCollectorManager.java | 15 +--
.../collector/TestNMTimelineCollectorManager.java | 6 --
3 files changed, 21 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 04a7003..20b96c2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -3809,6 +3809,10 @@ public class YarnConfiguration extends Configuration {
TIMELINE_SERVICE_COLLECTOR_PREFIX + "bind-host";
@Private
+ public static final String TIMELINE_SERVICE_COLLECTOR_BIND_PORT_RANGES =
+ TIMELINE_SERVICE_COLLECTOR_PREFIX + "bind-port-ranges";
+
+ @Private
public static final String TIMELINE_SERVICE_COLLECTOR_WEBAPP_ADDRESS =
TIMELINE_SERVICE_COLLECTOR_PREFIX + "webapp.address";
public static final String DEFAULT_TIMELINE_SERVICE_COLLECTOR_WEBAPP_ADDRESS
=
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/NodeTimelineCollectorManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/NodeTimelineCollectorManager.java
index 696f4a3..0371d49 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/NodeTimelineCollectorManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/NodeTimelineCollectorManager.java
@@ -58,6 +58,7 @@ import com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+
/**
* Class on the NodeManager side that manages adding and removing collectors
and
* their lifecycle. Also instantiates the per-node collector webapp.
@@ -280,14 +281,21 @@ public class NodeTimelineCollectorManager extends
TimelineCollectorManager {
String bindAddress = null;
String host =
conf.getTrimmed(YarnConfiguration.TIMELINE_SERVICE_COLLECTOR_BIND_HOST);
+Configuration.IntegerRanges portRanges = conf.getRange(
+YarnConfiguration.TIMELINE_SERVICE_COLLECTOR_BIND_PORT_RANGES, "");
+int startPort = 0;
+if (portRanges != null && !portRanges.isEmpty()) {
+ startPort = portRanges.getRangeStart();
+}
if (host == null || host.isEmpty()) {
// if collector bind-host is not set, fall back to
// timeline-service.bind-host to maintain compatibility
bindAddress =
conf.get(YarnConfiguration.DEFAULT_TIMELINE_SERVICE_BIND_HOST,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_BIND_HOST) + ":0";
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_BIND_HOST)
+ + ":" + startPort;
} else {
- bindAddress = host + ":0";
+ bindAddress = host + ":" + startPort;
}
try {
@@ -297,6 +305,9 @@ public class NodeTimelineCollectorManager extends
TimelineCollectorManager {
.addEndpoint(URI.create(
(YarnConfiguration.useHttps(conf) ? "https://; : "http://;) +
bindAddress));
+ if (portRanges != null && !portRanges.isEmpty()) {
+builder.setPortRanges(portRanges);
+ }
if (YarnConfiguration.useHttps(conf)) {
builder = WebAppUtils.loadSslConfiguration(builder, conf);
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestNMTimelineCollectorManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestNMTimelineCollectorManager.java
index af9acce..0f2af21 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-serve
[hadoop] 05/06: YARN-6735. Have a way to turn off container metrics from NMs. Contributed by Abhishek Modi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 108c569e3b6545872c80f5ba4a1644e65228363d
Author: Rohith Sharma K S
AuthorDate: Tue Feb 5 13:47:56 2019 +0530
YARN-6735. Have a way to turn off container metrics from NMs. Contributed
by Abhishek Modi.
---
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 10 +
.../src/main/resources/yarn-default.xml| 8 +
.../timelineservice/NMTimelinePublisher.java | 276 +++--
.../timelineservice/TestNMTimelinePublisher.java | 2 +
4 files changed, 168 insertions(+), 128 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 20b96c2..fe93fda 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1219,6 +1219,16 @@ public class YarnConfiguration extends Configuration {
public static final String DEFAULT_NM_COLLECTOR_SERVICE_ADDRESS =
"0.0.0.0:" + DEFAULT_NM_COLLECTOR_SERVICE_PORT;
+ /**
+ * The setting that controls whether yarn container events are published to
+ * the timeline service or not by NM. This configuration setting is for ATS
+ * V2
+ */
+ public static final String NM_PUBLISH_CONTAINER_EVENTS_ENABLED = NM_PREFIX
+ + "emit-container-events";
+ public static final boolean DEFAULT_NM_PUBLISH_CONTAINER_EVENTS_ENABLED =
+ true;
+
/** Interval in between cache cleanups.*/
public static final String NM_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS =
NM_PREFIX + "localizer.cache.cleanup.interval-ms";
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index f99977e..876dd6e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -1186,6 +1186,14 @@
+The setting that controls whether yarn container events are
+ published to the timeline service or not by NM. This configuration
setting
+ is for ATS V2.
+yarn.nodemanager.emit-container-events
+true
+
+
+
Interval in between cache cleanups.
yarn.nodemanager.localizer.cache.cleanup.interval-ms
60
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
index cbf3e5e..e9bd965 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
@@ -24,6 +24,7 @@ import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -88,6 +89,8 @@ public class NMTimelinePublisher extends CompositeService {
private final Map appToClientMap;
+ private boolean publishNMContainerEvents = true;
+
public NMTimelinePublisher(Context context) {
super(NMTimelinePublisher.class.getName());
this.context = context;
@@ -110,6 +113,10 @@ public class NMTimelinePublisher extends CompositeService {
if (webAppURLWithoutScheme.contains(":")) {
httpPort = webAppURLWithoutScheme.split(":")[1];
}
+
+publishNMContainerEvents = conf.getBoolean(
+YarnConfiguration.NM_PUBLISH_CONTAINER_EVENTS_ENABLED,
+YarnConfiguration.DEFAULT_NM_PUBLISH_CONTAINER_EVENTS_ENABLED);
super.serviceInit(conf);
}
@@ -155,31 +162,148 @@ public class NMTimelinePublisher extends
CompositeService {
public void reportContainerResourceUsage(Container container, Long pmemUsage,
Float cpuUsagePercentPerCore) {
-if (pmemUsage != ResourceCalculatorProcessTree.UNAVAILABLE ||
-cpuUsagePercentPerCore != ResourceCalculatorProcessTree.UNAVAILABLE) {
- Contain
[hadoop] 03/06: YARN-9034. ApplicationCLI should have option to take clusterId. Contributed by Rohith Sharma K S.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 0a6f90d4fc814355eea883818a646641d6a036e0
Author: Suma Shivaprasad
AuthorDate: Wed Nov 28 00:42:00 2018 -0800
YARN-9034. ApplicationCLI should have option to take clusterId. Contributed
by Rohith Sharma K S.
---
.../hadoop/yarn/client/cli/ApplicationCLI.java | 18 +
.../apache/hadoop/yarn/client/cli/ClusterCLI.java | 2 +
.../org/apache/hadoop/yarn/client/cli/NodeCLI.java | 2 +
.../apache/hadoop/yarn/client/cli/QueueCLI.java| 2 +-
.../org/apache/hadoop/yarn/client/cli/TopCLI.java | 1 +
.../org/apache/hadoop/yarn/client/cli/YarnCLI.java | 9 ++-
.../hadoop/yarn/client/cli/TestClusterCLI.java | 41 +-
.../apache/hadoop/yarn/client/cli/TestYarnCLI.java | 89 +++---
.../client/api/impl/TimelineReaderClientImpl.java | 2 +
9 files changed, 93 insertions(+), 73 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
index 480ea23..71f4c13 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
@@ -51,6 +51,7 @@ import org.apache.hadoop.yarn.api.records.Priority;
import org.apache.hadoop.yarn.api.records.SignalContainerCommand;
import org.apache.hadoop.yarn.api.records.YarnApplicationState;
import org.apache.hadoop.yarn.client.api.AppAdminClient;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException;
import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
import org.apache.hadoop.yarn.exceptions.ContainerNotFoundException;
@@ -111,6 +112,8 @@ public class ApplicationCLI extends YarnCLI {
public static final String COMPONENTS = "components";
public static final String VERSION = "version";
public static final String STATES = "states";
+ public static final String SHELL_CMD = "shell";
+ public static final String CLUSTER_ID_OPTION = "clusterId";
private static String firstArg = null;
@@ -276,6 +279,8 @@ public class ApplicationCLI extends YarnCLI {
"the ability to finalize the upgrade automatically.");
opts.addOption(UPGRADE_CANCEL, false, "Works with -upgrade option to " +
"cancel current upgrade.");
+ opts.addOption(CLUSTER_ID_OPTION, true, "ClusterId. "
+ + "By default, it will take default cluster id from the RM");
opts.getOption(LAUNCH_CMD).setArgName("Application Name> ());
-ClusterCLI cli = new ClusterCLI();
-cli.setClient(client);
-cli.setSysOutPrintStream(sysOut);
-cli.setSysErrPrintStream(sysErr);
+ClusterCLI cli = createAndGetClusterCLI();
int rc =
cli.run(new String[] { ClusterCLI.CMD, "-" +
ClusterCLI.LIST_LABELS_CMD });
@@ -154,9 +140,7 @@ public class TestClusterCLI {
@Test
public void testHelp() throws Exception {
-ClusterCLI cli = new ClusterCLI();
-cli.setSysOutPrintStream(sysOut);
-cli.setSysErrPrintStream(sysErr);
+ClusterCLI cli = createAndGetClusterCLI();
int rc =
cli.run(new String[] { "cluster", "--help" });
@@ -192,4 +176,15 @@ public class TestClusterCLI {
pw.close();
verify(sysOut).println(baos.toString("UTF-8"));
}
+
+ private ClusterCLI createAndGetClusterCLI() {
+ClusterCLI cli = new ClusterCLI() {
+ @Override protected void createAndStartYarnClient() {
+ }
+};
+cli.setClient(client);
+cli.setSysOutPrintStream(sysOut);
+cli.setSysErrPrintStream(sysErr);
+return cli;
+ }
}
\ No newline at end of file
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
index 9b1e863..459ed1e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
@@ -993,10 +993,7 @@ public class TestYarnCLI {
@Test (timeout = 5000)
public void testNodesHelpCommand() throws Exception {
-NodeCLI nodeCLI = new NodeCLI();
-nodeCLI.setClient(client);
-nodeCLI.setSysOutPrintStream(sysOut);
-nodeCLI.setSysErrPri
[hadoop] branch branch-3.2 updated (252afdc -> 84a9c3f)
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a change to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 252afdc YARN-9804. Update ATSv2 document for latest feature supports.
new a3496a3 YARN-8871. Document ATSv2 integrated LogWebService.
Contributed by Suma Shivaprasad.
new 4a4a892 YARN-7754. [Atsv2] Update document for running v1 and v2 TS.
Contributed by Suma Shivaprasad.
new 0a6f90d YARN-9034. ApplicationCLI should have option to take
clusterId. Contributed by Rohith Sharma K S.
new 5345508 YARN-6149. Allow port range to be specified while starting NM
Timeline collector manager. Contributed by Abhishek Modi.
new 108c569 YARN-6735. Have a way to turn off container metrics from NMs.
Contributed by Abhishek Modi.
new 84a9c3f YARN-5336 Limit the flow name size & consider cleanup for hex
chars. Contributed by Sushil Ks
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 27 ++
.../hadoop/yarn/client/cli/ApplicationCLI.java | 18 ++
.../apache/hadoop/yarn/client/cli/ClusterCLI.java | 2 +
.../org/apache/hadoop/yarn/client/cli/NodeCLI.java | 2 +
.../apache/hadoop/yarn/client/cli/QueueCLI.java| 2 +-
.../org/apache/hadoop/yarn/client/cli/TopCLI.java | 1 +
.../org/apache/hadoop/yarn/client/cli/YarnCLI.java | 9 +-
.../hadoop/yarn/client/cli/TestClusterCLI.java | 41 ++-
.../apache/hadoop/yarn/client/cli/TestYarnCLI.java | 89 ---
.../client/api/impl/TimelineReaderClientImpl.java | 2 +
.../hadoop/yarn/util/timeline/TimelineUtils.java | 32 +++
.../src/main/resources/yarn-default.xml| 19 ++
.../yarn/util/timeline/TestShortenedFlowName.java} | 46 ++--
.../timelineservice/NMTimelinePublisher.java | 276 +++--
.../timelineservice/TestNMTimelinePublisher.java | 2 +
.../server/timelineservice/TimelineContext.java| 9 +-
.../collector/NodeTimelineCollectorManager.java| 15 +-
.../collector/TestNMTimelineCollectorManager.java | 6 +-
.../src/site/markdown/TimelineServiceV2.md | 17 ++
19 files changed, 385 insertions(+), 230 deletions(-)
copy
hadoop-yarn-project/hadoop-yarn/{hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/api/TestServerRMProxy.java
=>
hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/timeline/TestShortenedFlowName.java}
(50%)
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 06/06: YARN-5336 Limit the flow name size & consider cleanup for hex chars. Contributed by Sushil Ks
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 84a9c3f999b72ceb04be8e2a10dc809f8be8871c
Author: Vrushali C
AuthorDate: Wed Feb 27 14:43:39 2019 -0800
YARN-5336 Limit the flow name size & consider cleanup for hex chars.
Contributed by Sushil Ks
---
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 13 ++
.../hadoop/yarn/util/timeline/TimelineUtils.java | 32 +
.../src/main/resources/yarn-default.xml| 11 +
.../yarn/util/timeline/TestShortenedFlowName.java | 52 ++
.../server/timelineservice/TimelineContext.java| 9 +++-
5 files changed, 115 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index fe93fda..ce33262 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -2713,6 +2713,19 @@ public class YarnConfiguration extends Configuration {
public static final String DEFAULT_HDFS_LOCATION_FLOW_RUN_COPROCESSOR_JAR =
"/hbase/coprocessor/hadoop-yarn-server-timelineservice.jar";
+ /**
+ * This setting controls the max size of the flow name getting generated
+ * in ATSv2 after removing UUID if present.
+ * */
+ public static final String FLOW_NAME_MAX_SIZE =
+ TIMELINE_SERVICE_PREFIX + "flowname.max-size";
+
+ /**
+ * Default setting for flow name size has no size restriction
+ * after removing UUID if present.
+ */
+ public static final int FLOW_NAME_DEFAULT_MAX_SIZE = 0;
+
/**
* The name for setting that points to an optional HBase configuration
* (hbase-site.xml file) with settings that will override the ones found on
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java
index 800e8ca..63a9ba5 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java
@@ -24,6 +24,8 @@ import java.net.InetSocketAddress;
import com.fasterxml.jackson.core.JsonGenerationException;
import com.fasterxml.jackson.databind.JsonMappingException;
import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.classification.InterfaceAudience.Public;
import org.apache.hadoop.classification.InterfaceStability.Evolving;
import org.apache.hadoop.conf.Configuration;
@@ -183,6 +185,36 @@ public class TimelineUtils {
}
/**
+ * Shortens the flow name for the configured size by removing UUID if
present.
+ *
+ * @param flowName which has to be shortened
+ * @param conf to resize the flow name
+ * @return shortened flowName
+ */
+ public static String shortenFlowName(String flowName, Configuration conf) {
+if (flowName == null) {
+ return null;
+}
+// remove UUID inside flowname if present
+flowName = removeUUID(flowName);
+// resize flowname
+int length = conf.getInt(YarnConfiguration.FLOW_NAME_MAX_SIZE,
+YarnConfiguration.FLOW_NAME_DEFAULT_MAX_SIZE);
+if (length <= 0) {
+ return flowName;
+}
+return StringUtils.substring(flowName, 0, length);
+ }
+
+ @VisibleForTesting
+ static String removeUUID(String flowName) {
+flowName = StringUtils.replaceAll(flowName,
+"-?([a-fA-F0-9]{8}-[a-fA-F0-9]{4}-" +
+"[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}){1}", "");
+return flowName;
+ }
+
+ /**
* Generate flow version tag.
*
* @param flowVersion flow version that keeps track of the changes made to
the
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index 876dd6e..ac7db9f 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -2624,6 +2624,17 @@
+
+
+ Removes the UUID if represent and limit the the flowname length with
+ the given value for ATSv2. In case the value is negative or 0,
+
[hadoop] 01/06: YARN-8871. Document ATSv2 integrated LogWebService. Contributed by Suma Shivaprasad.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit a3496a368bd2737393fedcfc34a9b1adb89c3926
Author: Rohith Sharma K S
AuthorDate: Tue Oct 30 11:34:20 2018 +0530
YARN-8871. Document ATSv2 integrated LogWebService. Contributed by Suma
Shivaprasad.
---
.../hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md | 5 +
1 file changed, 5 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
index 374d9bf..724bfd6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
@@ -1561,3 +1561,8 @@ With this API, you can query set of available entity
types for a given app id. I
1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is
returned.
1. If flow context information cannot be retrieved or entity for the given
entity id cannot be found, HTTP 404 (Not Found) is returned.
1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server
Error) is returned.
+
+## Aggregated Log
Serving for Historical Apps
+
+ TimelineService v.2 supports serving aggregated logs of historical apps. To
enable this, configure "yarn.log.server.web-service.url" to "${yarn
.timeline-service.hostname}:8188/ws/v2/applicationlog"
+ in `yarn-site.xml`
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 02/06: YARN-7754. [Atsv2] Update document for running v1 and v2 TS. Contributed by Suma Shivaprasad.
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/hadoop.git commit 4a4a892d328bf5a6d6993e990763bb088f0ca2d9 Author: Rohith Sharma K S AuthorDate: Tue Oct 30 11:35:01 2018 +0530 YARN-7754. [Atsv2] Update document for running v1 and v2 TS. Contributed by Suma Shivaprasad. --- .../hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md | 12 1 file changed, 12 insertions(+) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md index 724bfd6..457158d 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md @@ -324,6 +324,18 @@ that it can write data to the Apache HBase cluster you are using, or set ``` +To configure both Timeline Service 1.5 and v.2, add the following property + + ``` + + yarn.timeline-service.versions + 1.5f,2.0f + +``` + +If the above is not configured, then it defaults to the version set in `yarn.timeline-service.version` + + Running Timeline Service v.2 Restart the resource manager as well as the node managers to pick up the new configuration. The collectors start within the resource manager and the node managers in an embedded manner. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-9804. Update ATSv2 document for latest feature supports.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 252afdc YARN-9804. Update ATSv2 document for latest feature supports.
252afdc is described below
commit 252afdc8e6b3f5823b62a6309d9c86a501f0698f
Author: Rohith Sharma K S
AuthorDate: Thu Sep 5 08:42:17 2019 +0530
YARN-9804. Update ATSv2 document for latest feature supports.
---
.../src/site/markdown/TimelineServiceV2.md | 33 +++---
1 file changed, 16 insertions(+), 17 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
index 2ce1c32..374d9bf 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
@@ -73,8 +73,6 @@ The following diagram illustrates the design at a high level.
### Current Status and Future Plans
-YARN Timeline Service v.2 is currently in alpha ("alpha 2"). It is a work in
progress, and
-many things can and will change rapidly.
A complete end-to-end flow of writes and reads is functional, with Apache
HBase as the backend.
You should be able to start generating data. When enabled, all YARN-generic
events are
@@ -82,32 +80,31 @@ published as well as YARN system metrics such as CPU and
memory. Furthermore, so
including Distributed Shell and MapReduce can write per-framework data to YARN
Timeline Service
v.2.
-The basic mode of accessing data is via REST. Currently there is no support
for command line
-access. The REST API comes with a good number of useful and flexible query
patterns (see below for
-more information).
+The basic mode of accessing data is via REST. The REST API comes with a good
number of useful and flexible query patterns (see below for
+more information). YARN Client has been integrated with ATSv2. This enables
fetching application/attempt/container
+report from TimelineReader if details are not present in ResouceManager.
The collectors (writers) are currently embedded in the node managers as
auxiliary services. The
resource manager also has its dedicated in-process collector. The reader is
currently a single
instance. Currently, it is not possible to write to Timeline Service outside
the context of a YARN
application (i.e. no off-cluster client).
-Starting from alpha2, Timeline Service v.2 supports simple authorization in
terms of a
-configurable whitelist of users and groups who can read timeline data. Cluster
admins are
+Kerberos Authentication is supported end to end. All communication to HBase
can be kerberized. Refer [Security Configuration](#Security_Configuration) for
configs.
+Support for simple authorization has been added in terms of a configurable
whitelist of users and groups who can read timeline data. Cluster admins are
allowed by default to read timeline data.
+
When YARN Timeline Service v.2 is disabled, one can expect no functional or
performance impact
on any other existing functionality.
-The work to make it truly production-ready continues. Some key items include
-* More robust storage fault tolerance
+Road map includes
+
+* More robust storage fault tolerance.
* Support for off-cluster clients
-* Better support for long-running apps
-* Support for ACLs
+* Support for entity ACLs
* Offline (time-based periodic) aggregation for flows, users, and queues for
reporting and
analysis
-* Timeline collectors as separate instances from node managers
-* Clustering of the readers
* Migration and compatibility with v.1
@@ -142,8 +139,9 @@ New configuration parameters that are introduced with v.2
are marked bold.
|
**`yarn.timeline-service.timeline-client.number-of-async-entities-to-merge`** |
Time line V2 client tries to merge these many number of async entities (if
available) and then call the REST ATS V2 API to submit. Defaults to `10`. |
|
**`yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds`**
| The setting that controls how long the final value of a metric of a
completed app is retained before merging into the flow sum. Defaults to
`25920` (3 days). This should be set in the HBase cluster. |
| **`yarn.rm.system-metrics-publisher.emit-container-events`** | The setting
that controls whether yarn container metrics is published to the timeline
server or not by RM. This configuration setting is for ATS V2. Defaults to
`false`. |
+| **`yarn.nodemanager.emit-container-events`** | The setting that controls
whether yarn container metrics is published to the timeline server or not by
NM. This configuration setting is for ATS V2. Defaults to `true`. |
- Security Con
[hadoop] branch trunk updated: YARN-9804. Update ATSv2 document for latest feature supports.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 3db7184 YARN-9804. Update ATSv2 document for latest feature supports.
3db7184 is described below
commit 3db71840824c58344c2c59423fd605808785dc2c
Author: Rohith Sharma K S
AuthorDate: Thu Sep 5 08:42:17 2019 +0530
YARN-9804. Update ATSv2 document for latest feature supports.
---
.../src/site/markdown/TimelineServiceV2.md | 32 ++
1 file changed, 15 insertions(+), 17 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
index 16f14c1..35cbc58 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
@@ -73,8 +73,6 @@ The following diagram illustrates the design at a high level.
### Current Status and Future Plans
-YARN Timeline Service v.2 is currently in alpha ("alpha 2"). It is a work in
progress, and
-many things can and will change rapidly.
A complete end-to-end flow of writes and reads is functional, with Apache
HBase as the backend.
You should be able to start generating data. When enabled, all YARN-generic
events are
@@ -82,32 +80,31 @@ published as well as YARN system metrics such as CPU and
memory. Furthermore, so
including Distributed Shell and MapReduce can write per-framework data to YARN
Timeline Service
v.2.
-The basic mode of accessing data is via REST. Currently there is no support
for command line
-access. The REST API comes with a good number of useful and flexible query
patterns (see below for
-more information).
+The basic mode of accessing data is via REST. The REST API comes with a good
number of useful and flexible query patterns (see below for
+more information). YARN Client has been integrated with ATSv2. This enables
fetching application/attempt/container
+report from TimelineReader if details are not present in ResouceManager.
The collectors (writers) are currently embedded in the node managers as
auxiliary services. The
resource manager also has its dedicated in-process collector. The reader is
currently a single
instance. Currently, it is not possible to write to Timeline Service outside
the context of a YARN
application (i.e. no off-cluster client).
-Starting from alpha2, Timeline Service v.2 supports simple authorization in
terms of a
-configurable whitelist of users and groups who can read timeline data. Cluster
admins are
+Kerberos Authentication is supported end to end. All communication to HBase
can be kerberized. Refer [Security Configuration](#Security_Configuration) for
configs.
+Support for simple authorization has been added in terms of a configurable
whitelist of users and groups who can read timeline data. Cluster admins are
allowed by default to read timeline data.
+
When YARN Timeline Service v.2 is disabled, one can expect no functional or
performance impact
on any other existing functionality.
-The work to make it truly production-ready continues. Some key items include
-* More robust storage fault tolerance
+Road map includes
+
+* More robust storage fault tolerance.
* Support for off-cluster clients
-* Better support for long-running apps
-* Support for ACLs
+* Support for entity ACLs
* Offline (time-based periodic) aggregation for flows, users, and queues for
reporting and
analysis
-* Timeline collectors as separate instances from node managers
-* Clustering of the readers
* Migration and compatibility with v.1
@@ -143,7 +140,7 @@ New configuration parameters that are introduced with v.2
are marked bold.
|
**`yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds`**
| The setting that controls how long the final value of a metric of a
completed app is retained before merging into the flow sum. Defaults to
`25920` (3 days). This should be set in the HBase cluster. |
| **`yarn.rm.system-metrics-publisher.emit-container-events`** | The setting
that controls whether yarn container metrics is published to the timeline
server or not by RM. This configuration setting is for ATS V2. Defaults to
`false`. |
| **`yarn.nodemanager.emit-container-events`** | The setting that controls
whether yarn container metrics is published to the timeline server or not by
NM. This configuration setting is for ATS V2. Defaults to `true`. |
- Security Configuration
+ Security Configuration
Security can be enabled by setting
`yarn.timeline-service.http-authentication.type`
@@ -163,6 +160,7 @@ to `kerberos`, after which the following configuration
options are available:
| `yar
[hadoop] branch branch-3.2 updated: YARN-9714. ZooKeeper connection in ZKRMStateStore leaks after RM transitioned to standby. Contributed by Tao Yang.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 2fc4123 YARN-9714. ZooKeeper connection in ZKRMStateStore leaks after
RM transitioned to standby. Contributed by Tao Yang.
2fc4123 is described below
commit 2fc4123fe073ad48310bbe323206ab31d11de7ae
Author: Rohith Sharma K S
AuthorDate: Fri Aug 30 10:35:09 2019 +0530
YARN-9714. ZooKeeper connection in ZKRMStateStore leaks after RM
transitioned to standby. Contributed by Tao Yang.
---
.../hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java | 2 +-
.../hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
index bd76a8c..fb287fb 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
@@ -443,7 +443,7 @@ public class ZKRMStateStore extends RMStateStore {
verifyActiveStatusThread.join(1000);
}
-if (!HAUtil.isHAEnabled(getConfig())) {
+if (resourceManager.getZKManager() == null) {
CuratorFramework curatorFramework = zkManager.getCurator();
IOUtils.closeStream(curatorFramework);
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
index 11be3b1..c214672 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
@@ -450,6 +450,7 @@ public class TestZKRMStateStore extends
RMStateStoreTestBase {
rm = new MockRM(conf);
rm.start();
rm.getRMContext().getRMAdminService().transitionToActive(req);
+stateStore = (ZKRMStateStore) rm.getRMContext().getStateStore();
acls = stateStore.getACL(rootPath);
assertEquals(acls.size(), 1);
verifyZKACL("world", "anyone", Perms.ALL, acls);
@@ -460,6 +461,7 @@ public class TestZKRMStateStore extends
RMStateStoreTestBase {
rm = new MockRM(conf);
rm.start();
rm.getRMContext().getRMAdminService().transitionToActive(req);
+stateStore = (ZKRMStateStore) rm.getRMContext().getStateStore();
acls = stateStore.getACL(rootPath);
assertEquals(acls.size(), 2);
verifyZKACL("digest", "localhost", Perms.CREATE | Perms.DELETE, acls);
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-9714. ZooKeeper connection in ZKRMStateStore leaks after RM transitioned to standby. Contributed by Tao Yang.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 5b557de YARN-9714. ZooKeeper connection in ZKRMStateStore leaks after
RM transitioned to standby. Contributed by Tao Yang.
5b557de is described below
commit 5b557de721fa1587a7d0e197ea6b7120b53e8717
Author: Rohith Sharma K S
AuthorDate: Fri Aug 30 10:35:09 2019 +0530
YARN-9714. ZooKeeper connection in ZKRMStateStore leaks after RM
transitioned to standby. Contributed by Tao Yang.
---
.../hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java | 2 +-
.../hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
index dcf891a..fd0a214 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
@@ -453,7 +453,7 @@ public class ZKRMStateStore extends RMStateStore {
verifyActiveStatusThread.join(1000);
}
-if (!HAUtil.isHAEnabled(getConfig())) {
+if (resourceManager.getZKManager() == null) {
CuratorFramework curatorFramework = zkManager.getCurator();
IOUtils.closeStream(curatorFramework);
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
index f21ff96..6b6f013 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
@@ -453,6 +453,7 @@ public class TestZKRMStateStore extends
RMStateStoreTestBase {
rm = new MockRM(conf);
rm.start();
rm.getRMContext().getRMAdminService().transitionToActive(req);
+stateStore = (ZKRMStateStore) rm.getRMContext().getStateStore();
acls = stateStore.getACL(rootPath);
assertEquals(acls.size(), 1);
verifyZKACL("world", "anyone", Perms.ALL, acls);
@@ -463,6 +464,7 @@ public class TestZKRMStateStore extends
RMStateStoreTestBase {
rm = new MockRM(conf);
rm.start();
rm.getRMContext().getRMAdminService().transitionToActive(req);
+stateStore = (ZKRMStateStore) rm.getRMContext().getStateStore();
acls = stateStore.getACL(rootPath);
assertThat(acls).hasSize(2);
verifyZKACL("digest", "localhost", Perms.CREATE | Perms.DELETE, acls);
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-9796. Fix ASF license issue in branch-3.2. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.2 by this push: new 7616495 YARN-9796. Fix ASF license issue in branch-3.2. Contributed by Prabhu Joseph. 7616495 is described below commit 7616495fb72f6ca50612a43a46d8748597821d3a Author: Rohith Sharma K S AuthorDate: Thu Aug 29 10:49:01 2019 +0530 YARN-9796. Fix ASF license issue in branch-3.2. Contributed by Prabhu Joseph. --- .../node_modules/.bin/apidoc | 1 - .../node_modules/.bin/markdown-it | 1 - .../node_modules/.bin/r.js | 1 - .../node_modules/.bin/r_js | 1 - .../node_modules/.bin/semver | 1 - .../node_modules/.bin/shjs | 1 - .../yarn.lock | 422 - 7 files changed, 428 deletions(-) diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/apidoc b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/apidoc deleted file mode 12 index a588095..000 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/apidoc +++ /dev/null @@ -1 +0,0 @@ -../../target/generated-sources/vendor/apidoc/bin/apidoc \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/markdown-it b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/markdown-it deleted file mode 12 index 1ce5019..000 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/markdown-it +++ /dev/null @@ -1 +0,0 @@ -../../target/generated-sources/vendor/markdown-it/bin/markdown-it.js \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/r.js b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/r.js deleted file mode 12 index d4c664a..000 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/r.js +++ /dev/null @@ -1 +0,0 @@ -../../target/generated-sources/vendor/requirejs/bin/r.js \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/r_js b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/r_js deleted file mode 12 index d4c664a..000 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/r_js +++ /dev/null @@ -1 +0,0 @@ -../../target/generated-sources/vendor/requirejs/bin/r.js \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/semver b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/semver deleted file mode 12 index 2bb3fcb..000 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/semver +++ /dev/null @@ -1 +0,0 @@ -../../target/generated-sources/vendor/semver/bin/semver \ No newline at end of file diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/shjs b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/shjs deleted file mode 12 index 2928a9d..000 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/node_modules/.bin/shjs +++ /dev/null @@ -1 +0,0 @@ -../../target/generated-sources/vendor/shelljs/bin/shjs \ No newline at end of file diff --git
[hadoop] branch branch-3.2 updated: YARN-9640. Slow event processing could cause too many attempt unregister events. Contributed by Bibin A Chundatt.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 81c0809 YARN-9640. Slow event processing could cause too many attempt
unregister events. Contributed by Bibin A Chundatt.
81c0809 is described below
commit 81c08094634776c069f8595be7519af5681a9bd5
Author: Rohith Sharma K S
AuthorDate: Thu Aug 29 09:30:20 2019 +0530
YARN-9640. Slow event processing could cause too many attempt unregister
events. Contributed by Bibin A Chundatt.
---
.../resourcemanager/ApplicationMasterService.java | 14 --
.../TestApplicationMasterService.java | 54 ++
2 files changed, 65 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
index d9527d4..883da10 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
@@ -93,6 +93,8 @@ public class ApplicationMasterService extends AbstractService
implements
RecordFactoryProvider.getRecordFactory(null);
private final ConcurrentMap
responseMap =
new ConcurrentHashMap();
+ private final ConcurrentHashMap
+ finishedAttemptCache = new ConcurrentHashMap<>();
protected final RMContext rmContext;
private final AMSProcessingChain amsProcessingChain;
private boolean timelineServiceV2Enabled;
@@ -337,11 +339,14 @@ public class ApplicationMasterService extends
AbstractService implements
throw new ApplicationMasterNotRegisteredException(message);
}
- this.amLivelinessMonitor.receivedPing(applicationAttemptId);
FinishApplicationMasterResponse response =
FinishApplicationMasterResponse.newInstance(false);
- this.amsProcessingChain.finishApplicationMaster(
- applicationAttemptId, request, response);
+ if (finishedAttemptCache.putIfAbsent(applicationAttemptId, true)
+ == null) {
+this.amsProcessingChain
+.finishApplicationMaster(applicationAttemptId, request, response);
+ }
+ this.amLivelinessMonitor.receivedPing(applicationAttemptId);
return response;
}
}
@@ -490,6 +495,7 @@ public class ApplicationMasterService extends
AbstractService implements
public void unregisterAttempt(ApplicationAttemptId attemptId) {
LOG.info("Unregistering app attempt : " + attemptId);
responseMap.remove(attemptId);
+finishedAttemptCache.remove(attemptId);
rmContext.getNMTokenSecretManager().unregisterApplicationAttempt(attemptId);
}
@@ -504,6 +510,8 @@ public class ApplicationMasterService extends
AbstractService implements
if (this.server != null) {
this.server.stop();
}
+responseMap.clear();
+finishedAttemptCache.clear();
super.serviceStop();
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java
index 562ba5d..bfe908d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java
@@ -60,6 +60,9 @@ import org.apache.hadoop.yarn.api.records.ResourceInformation;
import org.apache.hadoop.yarn.api.records.ResourceRequest;
import org.apache.hadoop.yarn.api.records.UpdateContainerRequest;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.event.Dispatcher;
+import org.apache.hadoop.yarn.event.DrainDispatcher;
+import org.apache.hadoop.yarn.event.Event;
import
org.apache.hadoop.yarn.exceptions.ApplicationMasterNotRegisteredException;
import org.apache.hadoop.yarn.exceptions.InvalidContainerReleaseException;
import org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException;
@@ -70,6
[hadoop] branch trunk updated: YARN-9640. Slow event processing could cause too many attempt unregister events. Contributed by Bibin A Chundatt.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new d70f523 YARN-9640. Slow event processing could cause too many attempt
unregister events. Contributed by Bibin A Chundatt.
d70f523 is described below
commit d70f5231a794804d9baf0bdb7e6833533fe60db5
Author: Rohith Sharma K S
AuthorDate: Tue Aug 27 08:38:12 2019 +0530
YARN-9640. Slow event processing could cause too many attempt unregister
events. Contributed by Bibin A Chundatt.
---
.../resourcemanager/ApplicationMasterService.java | 14 --
.../ApplicationMasterServiceTestBase.java | 53 ++
2 files changed, 64 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
index a99da2c..31456d6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
@@ -94,6 +94,8 @@ public class ApplicationMasterService extends AbstractService
implements
RecordFactoryProvider.getRecordFactory(null);
private final ConcurrentMap
responseMap =
new ConcurrentHashMap();
+ private final ConcurrentHashMap
+ finishedAttemptCache = new ConcurrentHashMap<>();
protected final RMContext rmContext;
private final AMSProcessingChain amsProcessingChain;
private boolean timelineServiceV2Enabled;
@@ -339,11 +341,14 @@ public class ApplicationMasterService extends
AbstractService implements
throw new ApplicationMasterNotRegisteredException(message);
}
- this.amLivelinessMonitor.receivedPing(applicationAttemptId);
FinishApplicationMasterResponse response =
FinishApplicationMasterResponse.newInstance(false);
- this.amsProcessingChain.finishApplicationMaster(
- applicationAttemptId, request, response);
+ if (finishedAttemptCache.putIfAbsent(applicationAttemptId, true)
+ == null) {
+this.amsProcessingChain
+.finishApplicationMaster(applicationAttemptId, request, response);
+ }
+ this.amLivelinessMonitor.receivedPing(applicationAttemptId);
return response;
}
}
@@ -492,6 +497,7 @@ public class ApplicationMasterService extends
AbstractService implements
public void unregisterAttempt(ApplicationAttemptId attemptId) {
LOG.info("Unregistering app attempt : " + attemptId);
responseMap.remove(attemptId);
+finishedAttemptCache.remove(attemptId);
rmContext.getNMTokenSecretManager().unregisterApplicationAttempt(attemptId);
}
@@ -506,6 +512,8 @@ public class ApplicationMasterService extends
AbstractService implements
if (this.server != null) {
this.server.stop();
}
+responseMap.clear();
+finishedAttemptCache.clear();
super.serviceStop();
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterServiceTestBase.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterServiceTestBase.java
index 868b4e0..0b713e7 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterServiceTestBase.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterServiceTestBase.java
@@ -17,6 +17,10 @@
package org.apache.hadoop.yarn.server.resourcemanager;
import com.google.common.collect.ImmutableMap;
+import org.apache.hadoop.yarn.event.Dispatcher;
+import org.apache.hadoop.yarn.event.DrainDispatcher;
+import org.apache.hadoop.yarn.event.Event;
+import
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptEventType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse;
@@ -354,6 +358,55 @@ public abstract class ApplicationMasterServiceTestBase {
}
}
+ @Test(timeout = 120)
+ public void testRepeatedFinishApplicationMaster()
[hadoop] branch branch-3.2 updated: YARN-8917. Absolute (maximum) capacity of level3+ queues is wrongly calculated for absolute resource. Contributed by Tao Yang.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new ab98f91 YARN-8917. Absolute (maximum) capacity of level3+ queues is
wrongly calculated for absolute resource. Contributed by Tao Yang.
ab98f91 is described below
commit ab98f91638f38358cb71ace4d1570e01961c3f29
Author: Rohith Sharma K S
AuthorDate: Mon Aug 26 21:06:15 2019 +0530
YARN-8917. Absolute (maximum) capacity of level3+ queues is wrongly
calculated for absolute resource. Contributed by Tao Yang.
---
.../scheduler/capacity/ParentQueue.java| 8 ++--
.../TestAbsoluteResourceConfiguration.java | 49 ++
2 files changed, 53 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
index 0aefa69..443079f 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
@@ -1104,14 +1104,14 @@ public class ParentQueue extends AbstractCSQueue {
// 3. Update absolute capacity as a float based on parent's minResource and
// cluster resource.
childQueue.getQueueCapacities().setAbsoluteCapacity(label,
-(float) childQueue.getQueueCapacities().getCapacity()
-/ getQueueCapacities().getAbsoluteCapacity(label));
+childQueue.getQueueCapacities().getCapacity(label)
+* getQueueCapacities().getAbsoluteCapacity(label));
// 4. Update absolute max-capacity as a float based on parent's maxResource
// and cluster resource.
childQueue.getQueueCapacities().setAbsoluteMaximumCapacity(label,
-(float) childQueue.getQueueCapacities().getMaximumCapacity(label)
-/ getQueueCapacities().getAbsoluteMaximumCapacity(label));
+childQueue.getQueueCapacities().getMaximumCapacity(label)
+* getQueueCapacities().getAbsoluteMaximumCapacity(label));
// Re-visit max applications for a queue based on absolute capacity if
// needed.
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java
index 298e1ab..fe4c880 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java
@@ -33,6 +33,7 @@ import org.junit.Test;
public class TestAbsoluteResourceConfiguration {
private static final int GB = 1024;
+ private static final float DELTA = 0.001f;
private static final String QUEUEA = "queueA";
private static final String QUEUEB = "queueB";
@@ -544,6 +545,10 @@ public class TestAbsoluteResourceConfiguration {
QUEUE_A_MINRES, qA.queueResourceQuotas.getEffectiveMinResource());
Assert.assertEquals("Effective Max resource for QUEUEA is not correct",
QUEUE_A_MAXRES, qA.queueResourceQuotas.getEffectiveMaxResource());
+Assert.assertEquals("Absolute capacity for QUEUEA is not correct",
+0.4, qA.getAbsoluteCapacity(), DELTA);
+Assert.assertEquals("Absolute Max capacity for QUEUEA is not correct",
+0.8, qA.getAbsoluteMaximumCapacity(), DELTA);
ParentQueue qB = (ParentQueue) cs.getQueue(QUEUEB);
Assert.assertNotNull(qB);
@@ -555,6 +560,10 @@ public class TestAbsoluteResourceConfiguration {
QUEUE_B_MINRES, qB.queueResourceQuotas.getEffectiveMinResource());
Assert.assertEquals("Effective Max resource for QUEUEB is not correct",
QUEUE_B_MAXRES, qB.queueResourceQuotas.getEffectiveMaxResource());
+Assert.assertEquals("Absolute capacity for QUEUEB is not correct",
+0.2, qB.getAbsol
[hadoop] branch trunk updated: YARN-8917. Absolute (maximum) capacity of level3+ queues is wrongly calculated for absolute resource. Contributed by Tao Yang.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 689d2e6 YARN-8917. Absolute (maximum) capacity of level3+ queues is
wrongly calculated for absolute resource. Contributed by Tao Yang.
689d2e6 is described below
commit 689d2e61058b5f719c6cbe9897a72b19b44a29a3
Author: Rohith Sharma K S
AuthorDate: Mon Aug 26 21:06:15 2019 +0530
YARN-8917. Absolute (maximum) capacity of level3+ queues is wrongly
calculated for absolute resource. Contributed by Tao Yang.
---
.../scheduler/capacity/ParentQueue.java| 8 ++--
.../TestAbsoluteResourceConfiguration.java | 49 ++
2 files changed, 53 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
index c56369c..fb747fb 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
@@ -1102,14 +1102,14 @@ public class ParentQueue extends AbstractCSQueue {
// 3. Update absolute capacity as a float based on parent's minResource and
// cluster resource.
childQueue.getQueueCapacities().setAbsoluteCapacity(label,
-(float) childQueue.getQueueCapacities().getCapacity()
-/ getQueueCapacities().getAbsoluteCapacity(label));
+childQueue.getQueueCapacities().getCapacity(label)
+* getQueueCapacities().getAbsoluteCapacity(label));
// 4. Update absolute max-capacity as a float based on parent's maxResource
// and cluster resource.
childQueue.getQueueCapacities().setAbsoluteMaximumCapacity(label,
-(float) childQueue.getQueueCapacities().getMaximumCapacity(label)
-/ getQueueCapacities().getAbsoluteMaximumCapacity(label));
+childQueue.getQueueCapacities().getMaximumCapacity(label)
+* getQueueCapacities().getAbsoluteMaximumCapacity(label));
// Re-visit max applications for a queue based on absolute capacity if
// needed.
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java
index 298e1ab..fe4c880 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java
@@ -33,6 +33,7 @@ import org.junit.Test;
public class TestAbsoluteResourceConfiguration {
private static final int GB = 1024;
+ private static final float DELTA = 0.001f;
private static final String QUEUEA = "queueA";
private static final String QUEUEB = "queueB";
@@ -544,6 +545,10 @@ public class TestAbsoluteResourceConfiguration {
QUEUE_A_MINRES, qA.queueResourceQuotas.getEffectiveMinResource());
Assert.assertEquals("Effective Max resource for QUEUEA is not correct",
QUEUE_A_MAXRES, qA.queueResourceQuotas.getEffectiveMaxResource());
+Assert.assertEquals("Absolute capacity for QUEUEA is not correct",
+0.4, qA.getAbsoluteCapacity(), DELTA);
+Assert.assertEquals("Absolute Max capacity for QUEUEA is not correct",
+0.8, qA.getAbsoluteMaximumCapacity(), DELTA);
ParentQueue qB = (ParentQueue) cs.getQueue(QUEUEB);
Assert.assertNotNull(qB);
@@ -555,6 +560,10 @@ public class TestAbsoluteResourceConfiguration {
QUEUE_B_MINRES, qB.queueResourceQuotas.getEffectiveMinResource());
Assert.assertEquals("Effective Max resource for QUEUEB is not correct",
QUEUE_B_MAXRES, qB.queueResourceQuotas.getEffectiveMaxResource());
+Assert.assertEquals("Absolute capacity for QUEUEB is not correct",
+0.2, qB.getAbsol
[hadoop] 01/04: YARN-9389. FlowActivity and FlowRun table prefix is wrong. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit c1a4eeb7c877c78311044c2df4f810ae7c21489f
Author: Rohith Sharma K S
AuthorDate: Wed Mar 20 21:18:19 2019 +0530
YARN-9389. FlowActivity and FlowRun table prefix is wrong. Contributed by
Prabhu Joseph.
---
.../yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java | 2 +-
.../hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java
index 5b9fe13..ead6a79 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowActivityTableRW.java
@@ -36,7 +36,7 @@ import org.slf4j.LoggerFactory;
public class FlowActivityTableRW extends BaseTableRW {
/** flow activity table prefix. */
private static final String PREFIX =
- YarnConfiguration.TIMELINE_SERVICE_PREFIX + ".flowactivity";
+ YarnConfiguration.TIMELINE_SERVICE_PREFIX + "flowactivity";
/** config param name that specifies the flowactivity table name. */
public static final String TABLE_NAME_CONF_NAME = PREFIX + ".table.name";
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java
index 61c0734..b3e81a1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/flow/FlowRunTableRW.java
@@ -38,7 +38,7 @@ import org.apache.hadoop.hbase.Coprocessor;
public class FlowRunTableRW extends BaseTableRW {
/** entity prefix. */
private static final String PREFIX =
- YarnConfiguration.TIMELINE_SERVICE_PREFIX + ".flowrun";
+ YarnConfiguration.TIMELINE_SERVICE_PREFIX + "flowrun";
/** config param name that specifies the flowrun table name. */
public static final String TABLE_NAME_CONF_NAME = PREFIX + ".table.name";
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 02/04: YARN-9387. Update document for ATS HBase Custom tablenames (-entityTableName). Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit f6e60e966e3c843ae5e6777b8daf1ae411f4297f
Author: Rohith Sharma K S
AuthorDate: Wed Mar 20 21:20:50 2019 +0530
YARN-9387. Update document for ATS HBase Custom tablenames
(-entityTableName). Contributed by Prabhu Joseph.
---
.../src/site/markdown/TimelineServiceV2.md| 19 +++
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
index eb84377..cf98c57 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
@@ -284,10 +284,21 @@ The `TimelineSchemaCreator` tool supports a few options
that may come handy espe
are testing. For example, you can use `-skipExistingTable` (`-s` for short) to
skip existing tables
and continue to create other tables rather than failing the schema creation.
By default, the tables
will have a schema prefix of "prod.". When no option or '-help' ('-h' for
short) is provided, the
-command usage is printed.
-and continue to create other tables rather than failing the schema creation.
When no option or '-help'
-('-h' for short) is provided, the command usage is printed. By default, the
tables
-will have a schema prefix of "prod."
+command usage is printed. The options (-entityTableName, -appToflowTableName,
-applicationTableName,
+-subApplicationTableName) will help to override the default table names. On
using custom table names,
+The below corresponding configs with custom table name has to be set in
hbase-site.xml configured
+at yarn.timeline-service.hbase.configuration.file.
+
+```
+yarn.timeline-service.app-flow.table.name
+yarn.timeline-service.entity.table.name
+yarn.timeline-service.application.table.name
+yarn.timeline-service.subapplication.table.name
+yarn.timeline-service.flowactivity.table.name
+yarn.timeline-service.flowrun.table.name
+yarn.timeline-service.domain.table.name
+
+```
Enabling Timeline Service v.2
Following are the basic configurations to start Timeline service v.2:
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 04/04: YARN-9299. TestTimelineReaderWhitelistAuthorizationFilter ignores Http Errors. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit b3b0e332e6d023052d7bf96c7c1f82abab939e03
Author: Rohith Sharma K S
AuthorDate: Wed Mar 20 21:24:31 2019 +0530
YARN-9299. TestTimelineReaderWhitelistAuthorizationFilter ignores Http
Errors. Contributed by Prabhu Joseph.
---
...TimelineReaderWhitelistAuthorizationFilter.java | 58 +-
1 file changed, 46 insertions(+), 12 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWhitelistAuthorizationFilter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWhitelistAuthorizationFilter.java
index 9ab9b27..576699d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWhitelistAuthorizationFilter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWhitelistAuthorizationFilter.java
@@ -21,6 +21,8 @@ package org.apache.hadoop.yarn.server.timelineservice.reader;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
import java.io.IOException;
import java.security.Principal;
@@ -93,15 +95,19 @@ public class TestTimelineReaderWhitelistAuthorizationFilter
{
FilterConfig fc = new DummyFilterConfig(map);
f.init(fc);
HttpServletRequest mockHsr = Mockito.mock(HttpServletRequest.class);
+String userName = "user1";
Mockito.when(mockHsr.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
-return "user1";
+return userName;
}
});
HttpServletResponse r = Mockito.mock(HttpServletResponse.class);
f.doFilter(mockHsr, r, null);
+String msg = "User " + userName
++ " is not allowed to read TimelineService V2 data.";
+verify(r, times(0)).sendError(HttpServletResponse.SC_FORBIDDEN, msg);
}
@Test
@@ -143,15 +149,16 @@ public class
TestTimelineReaderWhitelistAuthorizationFilter {
FilterConfig fc = new DummyFilterConfig(map);
f.init(fc);
HttpServletRequest mockHsr = Mockito.mock(HttpServletRequest.class);
+String userName = "user1";
Mockito.when(mockHsr.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
-return "user1";
+return userName;
}
});
HttpServletResponse r = Mockito.mock(HttpServletResponse.class);
UserGroupInformation user1 =
-UserGroupInformation.createUserForTesting("user1", GROUP_NAMES);
+UserGroupInformation.createUserForTesting(userName, GROUP_NAMES);
user1.doAs(new PrivilegedExceptionAction() {
@Override
public Object run() throws Exception {
@@ -159,6 +166,9 @@ public class TestTimelineReaderWhitelistAuthorizationFilter
{
return null;
}
});
+String msg = "User " + userName
++ " is not allowed to read TimelineService V2 data.";
+verify(r, times(0)).sendError(HttpServletResponse.SC_FORBIDDEN, msg);
}
@Test
@@ -210,15 +220,16 @@ public class
TestTimelineReaderWhitelistAuthorizationFilter {
FilterConfig fc = new DummyFilterConfig(map);
f.init(fc);
HttpServletRequest mockHsr = Mockito.mock(HttpServletRequest.class);
+String userName = "user90";
Mockito.when(mockHsr.getUserPrincipal()).thenReturn(new Principal() {
@Override
public String getName() {
-return "user90";
+return userName;
}
});
HttpServletResponse r = Mockito.mock(HttpServletResponse.class);
UserGroupInformation user1 =
-UserGroupInformation.createUserForTesting("user90", GROUP_NAMES);
+UserGroupInformation.createUserForTesting(userName, GROUP_NAMES);
user1.doAs(new PrivilegedExceptionAction() {
@Override
public Object run() throws Exception {
@@ -226,6 +237,9 @@ public class TestTimelineReaderWhitelistAuthorizationFilter
{
return null;
}
});
+String msg = "User " + userName
++ " is not allowed to read TimelineService V2 data.";
+verify(r, times(0)).sendError(HttpServletResponse.SC_FORBIDDEN, msg);
}
@Test
@@ -240,15 +254,16 @@ public class
TestTime
[hadoop] branch trunk updated (072750c -> b3b0e33)
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 072750c HDDS-1306. TestContainerStateManagerIntegration fails in Ratis shutdown. Contributed by Lokesh Jain. new c1a4eeb YARN-9389. FlowActivity and FlowRun table prefix is wrong. Contributed by Prabhu Joseph. new f6e60e9 YARN-9387. Update document for ATS HBase Custom tablenames (-entityTableName). Contributed by Prabhu Joseph. new 0d24684 YARN-9357. Modify HBase Liveness monitor log to debug. Contributed by Prabhu Joseph. new b3b0e33 YARN-9299. TestTimelineReaderWhitelistAuthorizationFilter ignores Http Errors. Contributed by Prabhu Joseph. The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../storage/HBaseTimelineReaderImpl.java | 2 +- .../storage/flow/FlowActivityTableRW.java | 2 +- .../storage/flow/FlowRunTableRW.java | 2 +- ...TimelineReaderWhitelistAuthorizationFilter.java | 58 +- .../src/site/markdown/TimelineServiceV2.md | 19 +-- 5 files changed, 64 insertions(+), 19 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 03/04: YARN-9357. Modify HBase Liveness monitor log to debug. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 0d24684eee77dce7fd328d5e9479345e30b699ee
Author: Rohith Sharma K S
AuthorDate: Wed Mar 20 21:22:54 2019 +0530
YARN-9357. Modify HBase Liveness monitor log to debug. Contributed by
Prabhu Joseph.
---
.../yarn/server/timelineservice/storage/HBaseTimelineReaderImpl.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineReaderImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineReaderImpl.java
index f50621e..d00ae4b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineReaderImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/HBaseTimelineReaderImpl.java
@@ -167,7 +167,7 @@ public class HBaseTimelineReaderImpl
@Override
public void run() {
try {
-LOG.info("Running HBase liveness monitor");
+LOG.debug("Running HBase liveness monitor");
TimelineEntityReader reader =
TimelineEntityReaderFactory.createMultipleEntitiesReader(
monitorContext, MONITOR_FILTERS, DATA_TO_RETRIEVE);
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-9365. Fix table creation command in TimelineServiceV2.md
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new c9e50c4 YARN-9365. Fix table creation command in TimelineServiceV2.md
c9e50c4 is described below
commit c9e50c48de4ff7ef6bb89153362531833fd8da9f
Author: Rohith Sharma K S
AuthorDate: Tue Mar 19 12:05:28 2019 +0530
YARN-9365. Fix table creation command in TimelineServiceV2.md
---
.../src/site/markdown/TimelineServiceV2.md | 26 +-
1 file changed, 25 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
index 73176ef..eb84377 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
@@ -252,9 +252,33 @@ For example,
```
# Step 3) Create the timeline service schema
+The schema creation can be run on the hbase cluster which is going to store
the timeline
+service tables. The schema creator tool requires both the
timelineservice-hbase as well
+as the hbase-server jars. Hence, during schema creation, you need to ensure
that the
+hbase classpath contains the yarn-timelineservice-hbase jar.
+
+On the hbase cluster, you can get it from hdfs since we placed it there for the
+coprocessor in the step above.
+
+```
+ hadoop fs -get
/hbase/coprocessor/hadoop-yarn-server-timelineservice-hbase-client-${project.version}.jar
/.
+ hadoop fs -get
/hbase/coprocessor/hadoop-yarn-server-timelineservice-${project.version}.jar
/.
+ hadoop fs -get
/hbase/coprocessor/hadoop-yarn-server-timelineservice-hbase-common-${project.version}.jar
/.
+```
+
+Next, add it to the hbase classpath as follows:
+
+```
+ export
HBASE_CLASSPATH=$HBASE_CLASSPATH:/home/yarn/hadoop-current/share/hadoop/yarn/timelineservice/hadoop-yarn-server-timelineservice-hbase-client-${project.version}.jar
+ export
HBASE_CLASSPATH=$HBASE_CLASSPATH:/home/yarn/hadoop-current/share/hadoop/yarn/timelineservice/hadoop-yarn-server-timelineservice-${project.version}.jar
+ export
HBASE_CLASSPATH=$HBASE_CLASSPATH:/home/yarn/hadoop-current/share/hadoop/yarn/timelineservice/hadoop-yarn-server-timelineservice-hbase-common-${project.version}.jar
+```
+
Finally, run the schema creator tool to create the necessary tables:
-bin/hadoop
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
-create
+```
+bin/hbase
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator
-create
+```
The `TimelineSchemaCreator` tool supports a few options that may come handy
especially when you
are testing. For example, you can use `-skipExistingTable` (`-s` for short) to
skip existing tables
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-9336. JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController. Contributed by Tarun Parimi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new c24af4b YARN-9336. JobHistoryServer leaks CLOSE_WAIT tcp connections
when using LogAggregationIndexedFileController. Contributed by Tarun Parimi.
c24af4b is described below
commit c24af4b0d6fc32938b076161b5a8c86d38e3e0a1
Author: Rohith Sharma K S
AuthorDate: Tue Mar 12 20:57:27 2019 +0530
YARN-9336. JobHistoryServer leaks CLOSE_WAIT tcp connections when using
LogAggregationIndexedFileController. Contributed by Tarun Parimi.
---
.../ifile/IndexedFileAggregatedLogsBlock.java | 64 +++---
1 file changed, 32 insertions(+), 32 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/ifile/IndexedFileAggregatedLogsBlock.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/ifile/IndexedFileAggregatedLogsBlock.java
index eb9634b..c49d372 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/ifile/IndexedFileAggregatedLogsBlock.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/ifile/IndexedFileAggregatedLogsBlock.java
@@ -202,38 +202,38 @@ public class IndexedFileAggregatedLogsBlock extends
LogAggregationHtmlBlock {
Decompressor decompressor = compressName.getDecompressor();
FileContext fileContext = FileContext.getFileContext(
thisNodeFile.getPath().toUri(), conf);
-FSDataInputStream fsin = fileContext.open(thisNodeFile.getPath());
-int bufferSize = 65536;
-for (IndexedFileLogMeta candidate : candidates) {
- if (candidate.getLastModifiedTime() < startTime
- || candidate.getLastModifiedTime() > endTime) {
-continue;
- }
- byte[] cbuf = new byte[bufferSize];
- InputStream in = null;
- try {
-in = compressName.createDecompressionStream(
-new BoundedRangeFileInputStream(fsin, candidate.getStartIndex(),
-candidate.getFileCompressedSize()), decompressor,
-LogAggregationIndexedFileController.getFSInputBufferSize(conf));
-long logLength = candidate.getFileSize();
-html.pre().__("\n\n").__();
-html.p().__("Log Type: " + candidate.getFileName()).__();
-html.p().__(
-"Log Upload Time: " +
Times.format(candidate.getLastModifiedTime()))
-.__();
-html.p().__("Log Length: " + Long.toString(logLength)).__();
-
-long[] range = checkParseRange(html, start, end, startTime, endTime,
-logLength, candidate.getFileName());
-processContainerLog(html, range, in, bufferSize, cbuf);
-
-foundLog = true;
- } catch (Exception ex) {
-LOG.error("Error getting logs for " + logEntity, ex);
-continue;
- } finally {
-IOUtils.closeQuietly(in);
+try (FSDataInputStream fsin = fileContext.open(thisNodeFile.getPath())) {
+ int bufferSize = 65536;
+ for (IndexedFileLogMeta candidate : candidates) {
+if (candidate.getLastModifiedTime() < startTime
+|| candidate.getLastModifiedTime() > endTime) {
+ continue;
+}
+byte[] cbuf = new byte[bufferSize];
+InputStream in = null;
+try {
+ in = compressName.createDecompressionStream(
+ new BoundedRangeFileInputStream(fsin, candidate.getStartIndex(),
+ candidate.getFileCompressedSize()), decompressor,
+ LogAggregationIndexedFileController.getFSInputBufferSize(conf));
+ long logLength = candidate.getFileSize();
+ html.pre().__("\n\n").__();
+ html.p().__("Log Type: " + candidate.getFileName()).__();
+ html.p().__("Log Upload Time: " +
+ Times.format(candidate.getLastModifiedTime())).__();
+ html.p().__("Log Length: " + Long.toString(logLength)).__();
+
+ long[] range = checkParseRange(html, start, end, startTime, endTime,
+ logLength, candidate.getFileName());
+ processContainerLog(html, range, in, bufferSize, cbuf);
+
+ foundLog = true;
+} catch (Exception ex) {
+ LOG.error("Error getting logs for " + logEntity, ex);
+ continue;
+} finally {
+ IOUtils.closeQuietly(in);
+}
}
}
return foundLog;
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 01/02: YARN-9311. Fix TestRMRestart hangs. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 8eae260af50668cfdbb27d82136f17e367f7fde5
Author: Rohith Sharma K S
AuthorDate: Wed Feb 27 10:28:16 2019 +0530
YARN-9311. Fix TestRMRestart hangs. Contributed by Prabhu Joseph.
---
.../org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
index 7ebfe54..97883da 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
@@ -100,6 +100,7 @@ import
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore;
import
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.RMState;
import
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStoreAMRMTokenEvent;
import
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStoreEvent;
+import
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStoreProxyCAEvent;
import
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStoreRMDTEvent;
import
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStoreRMDTMasterKeyEvent;
import
org.apache.hadoop.yarn.server.resourcemanager.recovery.records.ApplicationAttemptStateData;
@@ -1646,7 +1647,8 @@ public class TestRMRestart extends
ParameterizedSchedulerTestBase {
// Skip if synchronous updation of DTToken
if (!(event instanceof RMStateStoreAMRMTokenEvent)
&& !(event instanceof RMStateStoreRMDTEvent)
-&& !(event instanceof RMStateStoreRMDTMasterKeyEvent)) {
+&& !(event instanceof RMStateStoreRMDTMasterKeyEvent)
+&& !(event instanceof RMStateStoreProxyCAEvent)) {
while (wait);
}
super.handleStoreEvent(event);
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated (625e937 -> 6c96f5e)
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 625e937 HDDS-1177. Add validation to AuthorizationHeaderV4. Contributed by Ajay Kumar new 8eae260 YARN-9311. Fix TestRMRestart hangs. Contributed by Prabhu Joseph. new 6c96f5e YARN-8378. ApplicationHistoryManagerImpl#getApplications doesn't honor filters. Contributed by Lantao Jin. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../ApplicationHistoryManagerImpl.java | 10 ++ .../ApplicationHistoryStoreTestUtils.java | 6 ++ .../TestApplicationHistoryManagerImpl.java | 21 + .../yarn/server/resourcemanager/TestRMRestart.java | 4 +++- 4 files changed, 40 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 02/02: YARN-8378. ApplicationHistoryManagerImpl#getApplications doesn't honor filters. Contributed by Lantao Jin.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 6c96f5e4b6aec05e9ce74bb229a317cdf95f5d40
Author: Rohith Sharma K S
AuthorDate: Wed Feb 27 10:32:58 2019 +0530
YARN-8378. ApplicationHistoryManagerImpl#getApplications doesn't honor
filters. Contributed by Lantao Jin.
---
.../ApplicationHistoryManagerImpl.java | 10 ++
.../ApplicationHistoryStoreTestUtils.java | 6 ++
.../TestApplicationHistoryManagerImpl.java | 21 +
3 files changed, 37 insertions(+)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerImpl.java
index 48a7294..385b806 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerImpl.java
@@ -104,10 +104,20 @@ public class ApplicationHistoryManagerImpl extends
AbstractService implements
historyStore.getAllApplications();
HashMap applicationsReport =
new HashMap();
+int count = 0;
for (Entry entry : histData
.entrySet()) {
+ if (count == appsNum) {
+break;
+ }
+ long appStartTime = entry.getValue().getStartTime();
+ if (appStartTime < appStartedTimeBegin
+ || appStartTime > appStartedTimeEnd) {
+continue;
+ }
applicationsReport.put(entry.getKey(),
convertToApplicationReport(entry.getValue()));
+ count++;
}
return applicationsReport;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryStoreTestUtils.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryStoreTestUtils.java
index de4051a..a6511de 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryStoreTestUtils.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryStoreTestUtils.java
@@ -47,6 +47,12 @@ public class ApplicationHistoryStoreTestUtils {
appId.toString(), "test type", "test queue", "test user", 0, 0));
}
+ protected void writeApplicationStartData(ApplicationId appId, long startTime)
+ throws IOException {
+store.applicationStarted(ApplicationStartData.newInstance(appId,
+appId.toString(), "test type", "test queue", "test user", 0,
startTime));
+ }
+
protected void writeApplicationFinishData(ApplicationId appId)
throws IOException {
store.applicationFinished(ApplicationFinishData.newInstance(appId, 0,
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerImpl.java
index fec2bf3..da5ddc1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryManagerImpl.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.yarn.server.applicationhistoryservice;
import java.io.IOException;
+import java.util.Map;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
@@ -71,4 +72,24 @@ public class TestApplicationHistoryManagerImpl extends
Assert.assertEquals("test type",
[hadoop] 01/02: YARN-7627. [ATSv2] When passing a non-number as metricslimit, the error message is wrong. Contributed by Charan Hebri.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 1e5e08d83b8dc168e636e0834584fecf5d44cb65
Author: Rohith Sharma K S
AuthorDate: Wed Feb 6 09:30:09 2019 +0530
YARN-7627. [ATSv2] When passing a non-number as metricslimit, the error
message is wrong. Contributed by Charan Hebri.
---
.../reader/TimelineReaderWebServices.java | 27 ++
1 file changed, 18 insertions(+), 9 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServices.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServices.java
index db48355..279cfdc 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServices.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServices.java
@@ -350,7 +350,8 @@ public class TimelineReaderWebServices {
succeeded = true;
} catch (Exception e) {
handleException(e, url, startTime,
- "createdTime start/end or limit or flowrunid");
+ "Either limit or createdtime start/end or metricslimit or
metricstime"
+ + " start/end or fromid");
} finally {
long latency = Time.monotonicNow() - startTime;
METRICS.addGetEntitiesLatency(latency, succeeded);
@@ -631,7 +632,8 @@ public class TimelineReaderWebServices {
succeeded = true;
} catch (Exception e) {
handleException(e, url, startTime,
- "createdTime start/end or limit or flowrunid");
+ "Either flowrunid or limit or createdtime start/end or metricslimit"
+ + " or metricstime start/end or fromid");
} finally {
long latency = Time.monotonicNow() - startTime;
METRICS.addGetEntitiesLatency(latency, succeeded);
@@ -725,7 +727,8 @@ public class TimelineReaderWebServices {
checkAccessForGenericEntity(entity, callerUGI);
succeeded = true;
} catch (Exception e) {
- handleException(e, url, startTime, "flowrunid");
+ handleException(e, url, startTime, "Either metricslimit or metricstime"
+ + " start/end");
} finally {
long latency = Time.monotonicNow() - startTime;
METRICS.addGetEntitiesLatency(latency, succeeded);
@@ -919,7 +922,8 @@ public class TimelineReaderWebServices {
checkAccessForGenericEntity(entity, callerUGI);
succeeded = true;
} catch (Exception e) {
- handleException(e, url, startTime, "flowrunid");
+ handleException(e, url, startTime, "Either flowrunid or metricslimit or"
+ + " metricstime start/end");
} finally {
long latency = Time.monotonicNow() - startTime;
METRICS.addGetEntitiesLatency(latency, succeeded);
@@ -1606,7 +1610,8 @@ public class TimelineReaderWebServices {
checkAccessForAppEntity(entity, callerUGI);
succeeded = true;
} catch (Exception e) {
- handleException(e, url, startTime, "flowrunid");
+ handleException(e, url, startTime, "Either metricslimit or metricstime"
+ + " start/end");
} finally {
long latency = Time.monotonicNow() - startTime;
METRICS.addGetEntitiesLatency(latency, succeeded);
@@ -1781,7 +1786,8 @@ public class TimelineReaderWebServices {
checkAccessForAppEntity(entity, callerUGI);
succeeded = true;
} catch (Exception e) {
- handleException(e, url, startTime, "flowrunid");
+ handleException(e, url, startTime, "Either flowrunid or metricslimit or"
+ + " metricstime start/end");
} finally {
long latency = Time.monotonicNow() - startTime;
METRICS.addGetEntitiesLatency(latency, succeeded);
@@ -1927,7 +1933,8 @@ public class TimelineReaderWebServices {
succeeded = true;
} catch (Exception e) {
handleException(e, url, startTime,
- "createdTime start/end or limit or flowrunid");
+ "Either limit or createdtime start/end or metricslimit or"
+ + " metricstime start/end");
} finally {
long latency = Time.monotonicNow() - startTime;
METRICS.addGetEntitiesLatency(latency, succeeded);
@@ -3418,7 +3425,8 @@ public class TimelineReaderWebServices {
succeeded = true;
} catch (Exception e) {
handleExce
[hadoop] branch trunk updated (912d9f7 -> 7fa62e1)
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 912d9f7 HDDS-1044. Client doesn't propogate correct error code to client on out of disk space. Contributed by Yiqun Lin. new 1e5e08d YARN-7627. [ATSv2] When passing a non-number as metricslimit, the error message is wrong. Contributed by Charan Hebri. new 7fa62e1 YARN-8219. Add application launch time to ATSV2. Contributed by Abhishek Modi. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../metrics/ApplicationMetricsConstants.java | 3 +++ .../util/timeline/TimelineEntityV2Converter.java | 9 ++-- .../metrics/AbstractSystemMetricsPublisher.java| 5 .../metrics/CombinedSystemMetricsPublisher.java| 6 + .../metrics/NoOpSystemMetricPublisher.java | 4 .../metrics/SystemMetricsPublisher.java| 2 ++ .../metrics/TimelineServiceV2Publisher.java| 14 +++ .../server/resourcemanager/rmapp/RMAppImpl.java| 4 +++- .../metrics/TestSystemMetricsPublisherForV2.java | 4 +++- .../reader/TimelineReaderWebServices.java | 27 ++ 10 files changed, 65 insertions(+), 13 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 02/02: YARN-8219. Add application launch time to ATSV2. Contributed by Abhishek Modi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 7fa62e150ce86f7cb0e88d593b8f07e3384a9a60
Author: Rohith Sharma K S
AuthorDate: Wed Feb 6 09:31:20 2019 +0530
YARN-8219. Add application launch time to ATSV2. Contributed by Abhishek
Modi.
---
.../yarn/server/metrics/ApplicationMetricsConstants.java | 3 +++
.../yarn/util/timeline/TimelineEntityV2Converter.java | 9 +++--
.../metrics/AbstractSystemMetricsPublisher.java| 5 +
.../metrics/CombinedSystemMetricsPublisher.java| 6 ++
.../resourcemanager/metrics/NoOpSystemMetricPublisher.java | 4
.../resourcemanager/metrics/SystemMetricsPublisher.java| 2 ++
.../metrics/TimelineServiceV2Publisher.java| 14 ++
.../yarn/server/resourcemanager/rmapp/RMAppImpl.java | 4 +++-
.../metrics/TestSystemMetricsPublisherForV2.java | 4 +++-
9 files changed, 47 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java
index 93da565..6428fa6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/server/metrics/ApplicationMetricsConstants.java
@@ -34,6 +34,9 @@ public class ApplicationMetricsConstants {
public static final String FINISHED_EVENT_TYPE =
"YARN_APPLICATION_FINISHED";
+ public static final String LAUNCHED_EVENT_TYPE =
+ "YARN_APPLICATION_LAUNCHED";
+
public static final String ACLS_UPDATED_EVENT_TYPE =
"YARN_APPLICATION_ACLS_UPDATED";
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineEntityV2Converter.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineEntityV2Converter.java
index 98e1e7a..9fefc09 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineEntityV2Converter.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineEntityV2Converter.java
@@ -258,6 +258,7 @@ public final class TimelineEntityV2Converter {
String type = null;
boolean unmanagedApplication = false;
long createdTime = 0;
+long launchTime = 0;
long finishedTime = 0;
float progress = 0.0f;
int applicationPriority = 0;
@@ -417,6 +418,9 @@ public final class TimelineEntityV2Converter {
ApplicationMetricsConstants.CREATED_EVENT_TYPE)) {
createdTime = event.getTimestamp();
} else if (event.getId().equals(
+ApplicationMetricsConstants.LAUNCHED_EVENT_TYPE)) {
+ launchTime = event.getTimestamp();
+} else if (event.getId().equals(
ApplicationMetricsConstants.UPDATED_EVENT_TYPE)) {
// This type of events are parsed in time-stamp descending order
// which means the previous event could override the information
@@ -449,8 +453,9 @@ public final class TimelineEntityV2Converter {
return ApplicationReport.newInstance(
ApplicationId.fromString(entity.getId()),
latestApplicationAttemptId, user, queue, name, null, -1, null, state,
-diagnosticsInfo, null, createdTime, finishedTime, finalStatus,
-appResources, null, progress, type, null, appTags,
unmanagedApplication,
+diagnosticsInfo, null, createdTime, launchTime,
+finishedTime, finalStatus, appResources, null,
+progress, type, null, appTags, unmanagedApplication,
Priority.newInstance(applicationPriority), appNodeLabelExpression,
amNodeLabelExpression);
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AbstractSystemMetricsPublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AbstractSystemMetricsPublisher.java
index 96b153f..09f44d0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AbstractSystemMetricsPublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/Abstr
[hadoop] branch trunk updated: YARN-6735. Have a way to turn off container metrics from NMs. Contributed by Abhishek Modi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new e3ec18b YARN-6735. Have a way to turn off container metrics from NMs.
Contributed by Abhishek Modi.
e3ec18b is described below
commit e3ec18b0c4fb93bd00c12d90057c2b982ca8a676
Author: Rohith Sharma K S
AuthorDate: Tue Feb 5 13:47:56 2019 +0530
YARN-6735. Have a way to turn off container metrics from NMs. Contributed
by Abhishek Modi.
---
.../apache/hadoop/yarn/conf/YarnConfiguration.java | 10 +
.../src/main/resources/yarn-default.xml| 8 +
.../timelineservice/NMTimelinePublisher.java | 276 +++--
.../timelineservice/TestNMTimelinePublisher.java | 2 +
.../src/site/markdown/TimelineServiceV2.md | 2 +-
5 files changed, 169 insertions(+), 129 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index de66e75..04d0fd1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1226,6 +1226,16 @@ public class YarnConfiguration extends Configuration {
public static final String DEFAULT_NM_COLLECTOR_SERVICE_ADDRESS =
"0.0.0.0:" + DEFAULT_NM_COLLECTOR_SERVICE_PORT;
+ /**
+ * The setting that controls whether yarn container events are published to
+ * the timeline service or not by NM. This configuration setting is for ATS
+ * V2
+ */
+ public static final String NM_PUBLISH_CONTAINER_EVENTS_ENABLED = NM_PREFIX
+ + "emit-container-events";
+ public static final boolean DEFAULT_NM_PUBLISH_CONTAINER_EVENTS_ENABLED =
+ true;
+
/** Interval in between cache cleanups.*/
public static final String NM_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS =
NM_PREFIX + "localizer.cache.cleanup.interval-ms";
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
index 45894e9..bfcbf4b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
@@ -1191,6 +1191,14 @@
+The setting that controls whether yarn container events are
+ published to the timeline service or not by NM. This configuration
setting
+ is for ATS V2.
+yarn.nodemanager.emit-container-events
+true
+
+
+
Interval in between cache cleanups.
yarn.nodemanager.localizer.cache.cleanup.interval-ms
60
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
index cbf3e5e..e9bd965 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java
@@ -24,6 +24,7 @@ import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -88,6 +89,8 @@ public class NMTimelinePublisher extends CompositeService {
private final Map appToClientMap;
+ private boolean publishNMContainerEvents = true;
+
public NMTimelinePublisher(Context context) {
super(NMTimelinePublisher.class.getName());
this.context = context;
@@ -110,6 +113,10 @@ public class NMTimelinePublisher extends CompositeService {
if (webAppURLWithoutScheme.contains(":")) {
httpPort = webAppURLWithoutScheme.split(":")[1];
}
+
+publishNMContainerEvents = conf.getBoolean(
+YarnConfiguration.NM_PUBLISH_CONTAINER_EVENTS_ENABLED,
+YarnConfiguration.DEFAULT_NM_PUBLISH_CONTAINER_EVENTS_ENABLED);
super.serviceInit(conf);
}
@@ -155,31 +162,148 @@ public class NMTimelinePublisher extends
CompositeService {
public void reportContaine
[hadoop] branch trunk updated: YARN-9149. yarn container -status misses logUrl when integrated with ATSv2. Contributed by Abhishek Modi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 5718389 YARN-9149. yarn container -status misses logUrl when
integrated with ATSv2. Contributed by Abhishek Modi.
5718389 is described below
commit 571838920d2faca703e3c3417d64123927dfc4ad
Author: Rohith Sharma K S
AuthorDate: Tue Feb 5 12:39:10 2019 +0530
YARN-9149. yarn container -status misses logUrl when integrated with ATSv2.
Contributed by Abhishek Modi.
---
.../apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java | 12 ++--
.../hadoop/yarn/client/api/impl/TestAHSv2ClientImpl.java | 9 +
.../hadoop/yarn/util/timeline/TimelineEntityV2Converter.java | 9 -
3 files changed, 27 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
index e797c28..19a7546 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
@@ -31,6 +31,7 @@ import org.apache.hadoop.yarn.api.records.ContainerReport;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity;
import org.apache.hadoop.yarn.client.api.AHSClient;
import org.apache.hadoop.yarn.client.api.TimelineReaderClient;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.util.timeline.TimelineEntityV2Converter;
@@ -48,6 +49,7 @@ import java.util.Map;
@InterfaceStability.Unstable
public class AHSv2ClientImpl extends AHSClient {
private TimelineReaderClient readerClient;
+ private String logServerUrl;
public AHSv2ClientImpl() {
super(AHSv2ClientImpl.class.getName());
@@ -55,6 +57,8 @@ public class AHSv2ClientImpl extends AHSClient {
@Override
public void serviceInit(Configuration conf) {
+logServerUrl = conf.get(
+YarnConfiguration.YARN_LOG_SERVER_URL);
readerClient = TimelineReaderClient.createTimelineReaderClient();
readerClient.init(conf);
}
@@ -119,15 +123,19 @@ public class AHSv2ClientImpl extends AHSClient {
@Override
public ContainerReport getContainerReport(ContainerId containerId)
throws YarnException, IOException {
+ApplicationReport appReport = getApplicationReport(
+containerId.getApplicationAttemptId().getApplicationId());
TimelineEntity entity = readerClient.getContainerEntity(containerId,
"ALL", null);
-return TimelineEntityV2Converter.convertToContainerReport(entity);
+return TimelineEntityV2Converter.convertToContainerReport(
+entity, logServerUrl, appReport.getUser());
}
@Override
public List getContainers(ApplicationAttemptId
applicationAttemptId) throws YarnException, IOException {
ApplicationId appId = applicationAttemptId.getApplicationId();
+ApplicationReport appReport = getApplicationReport(appId);
Map filters = new HashMap<>();
filters.put("infofilters", "SYSTEM_INFO_PARENT_ENTITY eq {\"id\":\"" +
applicationAttemptId.toString() +
@@ -140,7 +148,7 @@ public class AHSv2ClientImpl extends AHSClient {
for (TimelineEntity entity : entities) {
ContainerReport container =
TimelineEntityV2Converter.convertToContainerReport(
-entity);
+entity, logServerUrl, appReport.getUser());
containers.add(container);
}
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAHSv2ClientImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAHSv2ClientImpl.java
index 99473a3..1e0a1d6 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAHSv2ClientImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAHSv2ClientImpl.java
@@ -61,7 +61,10 @@ public class TestAHSv2ClientImpl {
Configuration conf = new YarnConfiguration();
conf.setBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED, true);
conf.setFloat(YarnConfiguration.TIMELINE_SERVICE_VERSION, 2.0f);
+conf.set(YarnConfiguration.YARN_LOG_SERVER_URL,
+"https://localhost:8188/ahs;);
client = new AHSv2ClientImpl();
+client.init(conf);
spyTimelineReaderClient = mock(
[hadoop] branch branch-3.1.2 updated: Revert "YARN-8270 Adding JMX Metrics for Timeline Collector and Reader. Contributed by Sushil Ks."
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.1.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1.2 by this push:
new 1019dde Revert "YARN-8270 Adding JMX Metrics for Timeline Collector
and Reader. Contributed by Sushil Ks."
1019dde is described below
commit 1019dde65bcf12e05ef48ac71e84550d589e5d9a
Author: Rohith Sharma K S
AuthorDate: Mon Jan 28 11:10:00 2019 +0530
Revert "YARN-8270 Adding JMX Metrics for Timeline Collector and Reader.
Contributed by Sushil Ks."
This reverts commit 5b72aa04e104242d1761abf56822fb38e9915def.
---
.../collector/TimelineCollectorWebService.java | 16 +-
.../PerNodeAggTimelineCollectorMetrics.java| 117 ---
.../metrics/TimelineReaderMetrics.java | 113 --
.../timelineservice/metrics/package-info.java | 28
.../reader/TimelineReaderWebServices.java | 166 +++--
.../TestPerNodeAggTimelineCollectorMetrics.java| 56 ---
.../reader/TestTimelineReaderMetrics.java | 56 ---
7 files changed, 52 insertions(+), 500 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
index 150249b..4e51fca 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
@@ -41,7 +41,6 @@ import
org.apache.hadoop.classification.InterfaceAudience.Public;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
import org.apache.hadoop.http.JettyUtils;
import org.apache.hadoop.security.UserGroupInformation;
-import org.apache.hadoop.util.Time;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import
org.apache.hadoop.yarn.api.records.timelineservice.ApplicationAttemptEntity;
import org.apache.hadoop.yarn.api.records.timelineservice.ApplicationEntity;
@@ -54,7 +53,6 @@ import
org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntityType;
import org.apache.hadoop.yarn.api.records.timelineservice.UserEntity;
-import
org.apache.hadoop.yarn.server.timelineservice.metrics.PerNodeAggTimelineCollectorMetrics;
import org.apache.hadoop.yarn.webapp.ForbiddenException;
import org.apache.hadoop.yarn.webapp.NotFoundException;
@@ -79,8 +77,6 @@ public class TimelineCollectorWebService {
LoggerFactory.getLogger(TimelineCollectorWebService.class);
private @Context ServletContext context;
- private static final PerNodeAggTimelineCollectorMetrics METRICS =
- PerNodeAggTimelineCollectorMetrics.getInstance();
/**
* Gives information about timeline collector.
@@ -155,15 +151,12 @@ public class TimelineCollectorWebService {
TimelineEntities entities) {
init(res);
UserGroupInformation callerUgi = getUser(req);
-boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
if (callerUgi == null) {
String msg = "The owner of the posted timeline entities is not set";
LOG.error(msg);
throw new ForbiddenException(msg);
}
-long startTime = Time.monotonicNow();
-boolean succeeded = false;
try {
ApplicationId appID = parseApplicationId(appId);
if (appID == null) {
@@ -178,6 +171,7 @@ public class TimelineCollectorWebService {
throw new NotFoundException("Application: "+ appId + " is not found");
}
+ boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
if (isAsync) {
collector.putEntitiesAsync(processTimelineEntities(entities, appId,
Boolean.valueOf(isSubAppEntities)), callerUgi);
@@ -186,7 +180,6 @@ public class TimelineCollectorWebService {
Boolean.valueOf(isSubAppEntities)), callerUgi);
}
- succeeded = true;
return Response.ok().build();
} catch (NotFoundException | ForbiddenException e) {
throw new WebApplicationException(e,
@@ -195,13 +188,6 @@ public class TimelineCollectorWebService {
LOG.error("Error putting entities", e);
throw new WebApplica
[hadoop] branch branch-3.1 updated: Revert "YARN-8270 Adding JMX Metrics for Timeline Collector and Reader. Contributed by Sushil Ks."
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 6e059c7 Revert "YARN-8270 Adding JMX Metrics for Timeline Collector
and Reader. Contributed by Sushil Ks."
6e059c7 is described below
commit 6e059c793069a1fc7d89ae2b3e680b634a3e0a81
Author: Rohith Sharma K S
AuthorDate: Mon Jan 28 10:55:12 2019 +0530
Revert "YARN-8270 Adding JMX Metrics for Timeline Collector and Reader.
Contributed by Sushil Ks."
This reverts commit 5b72aa04e104242d1761abf56822fb38e9915def.
---
.../collector/TimelineCollectorWebService.java | 16 +-
.../PerNodeAggTimelineCollectorMetrics.java| 117 ---
.../metrics/TimelineReaderMetrics.java | 113 --
.../timelineservice/metrics/package-info.java | 28
.../reader/TimelineReaderWebServices.java | 166 +++--
.../TestPerNodeAggTimelineCollectorMetrics.java| 56 ---
.../reader/TestTimelineReaderMetrics.java | 56 ---
7 files changed, 52 insertions(+), 500 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
index 150249b..4e51fca 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
@@ -41,7 +41,6 @@ import
org.apache.hadoop.classification.InterfaceAudience.Public;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
import org.apache.hadoop.http.JettyUtils;
import org.apache.hadoop.security.UserGroupInformation;
-import org.apache.hadoop.util.Time;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import
org.apache.hadoop.yarn.api.records.timelineservice.ApplicationAttemptEntity;
import org.apache.hadoop.yarn.api.records.timelineservice.ApplicationEntity;
@@ -54,7 +53,6 @@ import
org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntityType;
import org.apache.hadoop.yarn.api.records.timelineservice.UserEntity;
-import
org.apache.hadoop.yarn.server.timelineservice.metrics.PerNodeAggTimelineCollectorMetrics;
import org.apache.hadoop.yarn.webapp.ForbiddenException;
import org.apache.hadoop.yarn.webapp.NotFoundException;
@@ -79,8 +77,6 @@ public class TimelineCollectorWebService {
LoggerFactory.getLogger(TimelineCollectorWebService.class);
private @Context ServletContext context;
- private static final PerNodeAggTimelineCollectorMetrics METRICS =
- PerNodeAggTimelineCollectorMetrics.getInstance();
/**
* Gives information about timeline collector.
@@ -155,15 +151,12 @@ public class TimelineCollectorWebService {
TimelineEntities entities) {
init(res);
UserGroupInformation callerUgi = getUser(req);
-boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
if (callerUgi == null) {
String msg = "The owner of the posted timeline entities is not set";
LOG.error(msg);
throw new ForbiddenException(msg);
}
-long startTime = Time.monotonicNow();
-boolean succeeded = false;
try {
ApplicationId appID = parseApplicationId(appId);
if (appID == null) {
@@ -178,6 +171,7 @@ public class TimelineCollectorWebService {
throw new NotFoundException("Application: "+ appId + " is not found");
}
+ boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
if (isAsync) {
collector.putEntitiesAsync(processTimelineEntities(entities, appId,
Boolean.valueOf(isSubAppEntities)), callerUgi);
@@ -186,7 +180,6 @@ public class TimelineCollectorWebService {
Boolean.valueOf(isSubAppEntities)), callerUgi);
}
- succeeded = true;
return Response.ok().build();
} catch (NotFoundException | ForbiddenException e) {
throw new WebApplicationException(e,
@@ -195,13 +188,6 @@ public class TimelineCollectorWebService {
LOG.error("Error putting entities", e);
throw new WebApplica
[hadoop] branch trunk updated: YARN-7761. [UI2] Clicking 'master container log' or 'Link' next to 'log' under application's appAttempt goes to Old UI's Log link. Contributed by Akhil PB.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 84bb980 YARN-7761. [UI2] Clicking 'master container log' or 'Link'
next to 'log' under application's appAttempt goes to Old UI's Log link.
Contributed by Akhil PB.
84bb980 is described below
commit 84bb9808a2f56fcdfbfe496337033d9aa0cb583e
Author: Rohith Sharma K S
AuthorDate: Fri Jan 25 14:00:19 2019 +0530
YARN-7761. [UI2] Clicking 'master container log' or 'Link' next to 'log'
under application's appAttempt goes to Old UI's Log link. Contributed by Akhil
PB.
---
.../src/main/webapp/app/components/timeline-view.js | 12 ++--
.../src/main/webapp/app/controllers/yarn-app/logs.js | 9 +++--
.../src/main/webapp/app/helpers/log-files-comma.js | 2 +-
.../src/main/webapp/app/initializers/loader.js | 4 +++-
.../src/main/webapp/app/models/yarn-app-attempt.js | 6 ++
.../src/main/webapp/app/models/yarn-container.js | 9 -
.../src/main/webapp/app/models/yarn-timeline-container.js| 9 -
.../src/main/webapp/app/routes/yarn-app/logs.js | 6 ++
.../src/main/webapp/app/routes/yarn-node-container.js| 8 +---
.../webapp/app/templates/components/app-attempt-table.hbs| 8 +---
.../main/webapp/app/templates/components/container-table.hbs | 8 +---
.../src/main/webapp/app/templates/yarn-app.hbs | 4 ++--
.../src/main/webapp/app/templates/yarn-apps/apps.hbs | 2 ++
.../src/main/webapp/app/templates/yarn-node/info.hbs | 4 +---
14 files changed, 65 insertions(+), 26 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/components/timeline-view.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/components/timeline-view.js
index b4de94f..031dd41 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/components/timeline-view.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/components/timeline-view.js
@@ -408,9 +408,9 @@ export default Ember.Component.extend({
contentPath: 'logsLink',
cellComponentName: 'em-table-html-cell',
getCellContent: function(row) {
-var logUrl = self.checkHttpProtocol(row.get('logsLink'));
-if (logUrl) {
- return `Link`;
+var containerLogUrl = row.get('appAttemptContainerLogsURL');
+if (containerLogUrl) {
+ return `Link`;
} else {
return 'N/A';
}
@@ -490,9 +490,9 @@ export default Ember.Component.extend({
contentPath: 'logUrl',
cellComponentName: 'em-table-html-cell',
getCellContent: function(row) {
-var url = self.checkHttpProtocol(row.get('logUrl'));
-if (url) {
- return `${url}`;
+var containerLogUrl = row.get('appAttemptContainerLogsURL');
+if (containerLogUrl) {
+ return `Link`;
} else {
return 'N/A';
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-app/logs.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-app/logs.js
index 8224011..e1a8ec8 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-app/logs.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-app/logs.js
@@ -20,8 +20,10 @@ import Ember from 'ember';
import Constants from 'yarn-ui/constants';
export default Ember.Controller.extend({
- queryParams: ["service"],
+ queryParams: ["service", "attempt", "containerid"],
service: undefined,
+ attempt: undefined,
+ containerid: undefined,
selectedAttemptId: "",
attemptContainerList: null,
@@ -40,7 +42,7 @@ export default Ember.Controller.extend({
},
actions: {
-showContainersForAttemptId(attemptId) {
+showContainersForAttemptId(attemptId, containerId = "") {
this.set("selectedAttemptId", "");
if (attemptId) {
this.set("_isLoadingTopPanel", true);
@@ -75,6 +77,9 @@ export default Ember.Controller.extend({
}
this.set("attemptContainerList", containers);
this.initializeSelect(".js-fetch-logs-containers");
+if (containerId) {
+ this.send("showLogFilesForContainerId", containerId);
+}
})
.finally(() => {
this.set("_isLoadingTopPanel", false);
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/helpers/log-files-comma.js
b/hadoop-yarn-project/hado
[hadoop] 01/02: YARN-8567. Fetching yarn logs fails for long running application if it is not present in timeline store. Contributed by Tarun Parimi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 573b1587918c4c0efdb7e9fff6f5be12bf31b619
Author: Rohith Sharma K S
AuthorDate: Fri Jan 4 14:01:24 2019 +0530
YARN-8567. Fetching yarn logs fails for long running application if it is
not present in timeline store. Contributed by Tarun Parimi.
---
.../yarn/client/api/impl/YarnClientImpl.java | 2 +-
.../yarn/client/api/impl/TestYarnClient.java | 50 ++
2 files changed, 51 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
index 28a7f85..86bda98 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
@@ -879,7 +879,7 @@ public class YarnClientImpl extends YarnClient {
try {
containersListFromAHS =
getContainerReportFromHistory(applicationAttemptId);
-} catch (IOException e) {
+} catch (IOException | YarnException e) {
if (appNotFoundInRM) {
throw e;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
index 17e43ca..7ad7e3a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
@@ -531,6 +531,44 @@ public class TestYarnClient extends
ParameterizedSchedulerTestBase {
}
@Test(timeout = 1)
+ public void testGetContainersOnAHSFail() throws YarnException, IOException {
+Configuration conf = getConf();
+conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+true);
+
+final YarnClient client = new MockYarnClient() {
+ @Override
+ public List getContainers(
+ ApplicationAttemptId appAttemptId) throws YarnException,
+ IOException {
+return getContainersOnAHSFail(appAttemptId);
+ }
+};
+
+client.init(conf);
+client.start();
+
+ApplicationId applicationId = ApplicationId.newInstance(1234, 5);
+ApplicationAttemptId appAttemptId = ApplicationAttemptId.newInstance(
+applicationId, 1);
+List reports = client.getContainers(appAttemptId);
+Assert.assertNotNull(reports);
+Assert.assertTrue(reports.size() == 2);
+Assert.assertEquals(reports.get(0).getContainerId(),
+(ContainerId.newContainerId(appAttemptId, 1)));
+Assert.assertEquals(reports.get(1).getContainerId(),
+(ContainerId.newContainerId(appAttemptId, 2)));
+
+//Only 2 running containers from RM are present when AHS throws exception
+Assert.assertEquals(ContainerState.RUNNING,
+(reports.get(0).getContainerState()));
+Assert.assertEquals(ContainerState.RUNNING,
+(reports.get(1).getContainerState()));
+client.stop();
+ }
+
+
+ @Test(timeout = 1)
public void testGetContainerReport() throws YarnException, IOException {
Configuration conf = getConf();
conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
@@ -914,6 +952,18 @@ public class TestYarnClient extends
ParameterizedSchedulerTestBase {
return super.getContainers(appAttemptId);
}
+protected List
+getContainersOnAHSFail(ApplicationAttemptId appAttemptId)
+ throws YarnException, IOException {
+ when(mockContainersResponse.getContainerList()).thenReturn(
+ getContainersReport(appAttemptId));
+ when(historyClient.getContainers(any(ApplicationAttemptId.class)))
+ .thenThrow(new ApplicationNotFoundException(
+ appAttemptId.getApplicationId() +
+ " does not exist in the timeline store"));
+ return super.getContainers(appAttemptId);
+}
+
private List getContainersFromAHS(
ApplicationAttemptId appAttemptId) {
return containersFromAHS.get(appAttemptId);
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 02/02: YARN-6149. Allow port range to be specified while starting NM Timeline collector manager. Contributed by Abhishek Modi.
This is an automated email from the ASF dual-hosted git repository.
rohithsharmaks pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 8c6978c3baef96a333ebd7e98e02098c99df7313
Author: Rohith Sharma K S
AuthorDate: Fri Jan 4 14:07:02 2019 +0530
YARN-6149. Allow port range to be specified while starting NM Timeline
collector manager. Contributed by Abhishek Modi.
---
.../org/apache/hadoop/yarn/conf/YarnConfiguration.java| 4
.../collector/NodeTimelineCollectorManager.java | 15 +--
.../collector/TestNMTimelineCollectorManager.java | 6 --
3 files changed, 21 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 511edef..08f7f1a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -3869,6 +3869,10 @@ public class YarnConfiguration extends Configuration {
TIMELINE_SERVICE_COLLECTOR_PREFIX + "bind-host";
@Private
+ public static final String TIMELINE_SERVICE_COLLECTOR_BIND_PORT_RANGES =
+ TIMELINE_SERVICE_COLLECTOR_PREFIX + "bind-port-ranges";
+
+ @Private
public static final String TIMELINE_SERVICE_COLLECTOR_WEBAPP_ADDRESS =
TIMELINE_SERVICE_COLLECTOR_PREFIX + "webapp.address";
public static final String DEFAULT_TIMELINE_SERVICE_COLLECTOR_WEBAPP_ADDRESS
=
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/NodeTimelineCollectorManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/NodeTimelineCollectorManager.java
index 696f4a3..0371d49 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/NodeTimelineCollectorManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/NodeTimelineCollectorManager.java
@@ -58,6 +58,7 @@ import com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+
/**
* Class on the NodeManager side that manages adding and removing collectors
and
* their lifecycle. Also instantiates the per-node collector webapp.
@@ -280,14 +281,21 @@ public class NodeTimelineCollectorManager extends
TimelineCollectorManager {
String bindAddress = null;
String host =
conf.getTrimmed(YarnConfiguration.TIMELINE_SERVICE_COLLECTOR_BIND_HOST);
+Configuration.IntegerRanges portRanges = conf.getRange(
+YarnConfiguration.TIMELINE_SERVICE_COLLECTOR_BIND_PORT_RANGES, "");
+int startPort = 0;
+if (portRanges != null && !portRanges.isEmpty()) {
+ startPort = portRanges.getRangeStart();
+}
if (host == null || host.isEmpty()) {
// if collector bind-host is not set, fall back to
// timeline-service.bind-host to maintain compatibility
bindAddress =
conf.get(YarnConfiguration.DEFAULT_TIMELINE_SERVICE_BIND_HOST,
- YarnConfiguration.DEFAULT_TIMELINE_SERVICE_BIND_HOST) + ":0";
+ YarnConfiguration.DEFAULT_TIMELINE_SERVICE_BIND_HOST)
+ + ":" + startPort;
} else {
- bindAddress = host + ":0";
+ bindAddress = host + ":" + startPort;
}
try {
@@ -297,6 +305,9 @@ public class NodeTimelineCollectorManager extends
TimelineCollectorManager {
.addEndpoint(URI.create(
(YarnConfiguration.useHttps(conf) ? "https://; : "http://;) +
bindAddress));
+ if (portRanges != null && !portRanges.isEmpty()) {
+builder.setPortRanges(portRanges);
+ }
if (YarnConfiguration.useHttps(conf)) {
builder = WebAppUtils.loadSslConfiguration(builder, conf);
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestNMTimelineCollectorManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestNMTimelineCollectorManager.java
index af9acce..0f2af21 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-serve
[hadoop] branch trunk updated (f4906ac -> 8c6978c)
This is an automated email from the ASF dual-hosted git repository. rohithsharmaks pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from f4906ac YARN-9038. [CSI] Add ability to publish/unpublish volumes on node managers. Contributed by Weiwei Yang. new 573b158 YARN-8567. Fetching yarn logs fails for long running application if it is not present in timeline store. Contributed by Tarun Parimi. new 8c6978c YARN-6149. Allow port range to be specified while starting NM Timeline collector manager. Contributed by Abhishek Modi. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/hadoop/yarn/conf/YarnConfiguration.java | 4 ++ .../yarn/client/api/impl/YarnClientImpl.java | 2 +- .../yarn/client/api/impl/TestYarnClient.java | 50 ++ .../collector/NodeTimelineCollectorManager.java| 15 ++- .../collector/TestNMTimelineCollectorManager.java | 6 ++- 5 files changed, 72 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-9042. Fix javadoc error in deviceplugin package. Contributed by Zhankun Tang.
Repository: hadoop
Updated Branches:
refs/heads/trunk 95d526a27 -> e223a790a
YARN-9042. Fix javadoc error in deviceplugin package. Contributed by Zhankun
Tang.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e223a790
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e223a790
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e223a790
Branch: refs/heads/trunk
Commit: e223a790a7605721c5226d09e561ca469fa7d7bf
Parents: 95d526a
Author: Rohith Sharma K S
Authored: Thu Nov 22 17:39:06 2018 +0530
Committer: Rohith Sharma K S
Committed: Thu Nov 22 20:04:30 2018 +0530
--
.../hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java | 3 ++-
.../yarn/server/nodemanager/api/deviceplugin/DevicePlugin.java | 4
.../server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java | 4 ++--
3 files changed, 8 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e223a790/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java
index 551c2d8..c3a2515 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java
@@ -53,7 +53,8 @@ public final class Device implements Serializable, Comparable
{
private final int minorNumber;
/**
- * PCI Bus ID in format ]:]]:][][.[]].
+ * PCI Bus ID in format.
+ * domain]:]bus]:][slot][.[func]].
* Optional. Can get from "lspci -D" in Linux
* */
private final String busID;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e223a790/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DevicePlugin.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DevicePlugin.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DevicePlugin.java
index 5b7a569..e93ec4d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DevicePlugin.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DevicePlugin.java
@@ -27,6 +27,7 @@ public interface DevicePlugin {
/**
* Called first when device plugin framework wants to register.
* @return DeviceRegisterRequest {@link DeviceRegisterRequest}
+ * @throws Exception
* */
DeviceRegisterRequest getRegisterRequestInfo()
throws Exception;
@@ -34,6 +35,7 @@ public interface DevicePlugin {
/**
* Called when update node resource.
* @return a set of {@link Device}, {@link java.util.TreeSet} recommended
+ * @throws Exception
* */
Set getDevices() throws Exception;
@@ -52,6 +54,7 @@ public interface DevicePlugin {
*docker container runtime is used.
* @return a {@link DeviceRuntimeSpec} description about environment,
* {@link VolumeSpec}, {@link MountVolumeSpec}. etc
+ * @throws Exception
* */
DeviceRuntimeSpec onDevicesAllocated(Set allocatedDevices,
YarnRuntimeType yarnRuntime) throws Exception;
@@ -59,6 +62,7 @@ public interface DevicePlugin {
/**
* Called after device released.
* @param releasedDevices A set of released devices
+ * @throws Exception
* */
void onDevicesReleased(Set releasedDevices)
throws Exception;
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e223a790/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java
--
diff
hadoop git commit: YARN-8936. Bump up Atsv2 hbase versions. Contributed by Vrushali C.
Repository: hadoop Updated Branches: refs/heads/trunk c8b3dfa62 -> f63e4e4f2 YARN-8936. Bump up Atsv2 hbase versions. Contributed by Vrushali C. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f63e4e4f Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f63e4e4f Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f63e4e4f Branch: refs/heads/trunk Commit: f63e4e4f2eee6b75c3a6c3fb5926d44c7e846145 Parents: c8b3dfa Author: Rohith Sharma K S Authored: Wed Nov 21 15:02:22 2018 +0530 Committer: Rohith Sharma K S Committed: Wed Nov 21 15:02:22 2018 +0530 -- hadoop-project/pom.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/f63e4e4f/hadoop-project/pom.xml -- diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 6f83280..a205d2b 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -154,8 +154,8 @@ 1.5.4 1.16 -1.2.6 -2.0.0-beta-1 +1.4.8 +2.0.2 4.12 5.3.1 5.3.1 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[1/2] hadoop git commit: YARN-8834 Provide Java client for fetching Yarn specific entities from TimelineReader. Contributed by Abhishek Modi
Repository: hadoop
Updated Branches:
refs/heads/branch-3.2 7a2a80621 -> 13e3670e7
YARN-8834 Provide Java client for fetching Yarn specific entities from
TimelineReader. Contributed by Abhishek Modi
(cherry picked from commit a3edfddcf7822ea13bdf4858672eb82cea5e0b5f)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/da053288
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/da053288
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/da053288
Branch: refs/heads/branch-3.2
Commit: da05328824a61eb3b35d7eab9f721e3172700f11
Parents: 7a2a806
Author: Vrushali C
Authored: Thu Oct 11 20:21:00 2018 -0700
Committer: Rohith Sharma K S
Committed: Fri Nov 16 18:37:01 2018 +0530
--
.../yarn/client/api/TimelineReaderClient.java | 120 ++
.../api/impl/TimelineReaderClientImpl.java | 239 +++
.../api/impl/TestTimelineReaderClientImpl.java | 157
3 files changed, 516 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/da053288/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineReaderClient.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineReaderClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineReaderClient.java
new file mode 100644
index 000..f73c2d3
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineReaderClient.java
@@ -0,0 +1,120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.client.api;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity;
+import org.apache.hadoop.yarn.client.api.impl.TimelineReaderClientImpl;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A client library that can be used to get Timeline Entities associated with
+ * application, application attempt or containers. This client library needs to
+ * be used along with time line v.2 server version.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public abstract class TimelineReaderClient extends CompositeService {
+
+ /**
+ * Create a new instance of Timeline Reader Client.
+ */
+ @InterfaceAudience.Public
+ public static TimelineReaderClient createTimelineReaderClient() {
+return new TimelineReaderClientImpl();
+ }
+
+ @InterfaceAudience.Private
+ public TimelineReaderClient(String name) {
+super(name);
+ }
+
+ /**
+ * Gets application entity.
+ * @param appId application id
+ * @param fields Fields to be fetched. Defaults to INFO.
+ * @param filters Filters to be applied while fetching entities.
+ * @return entity of the application
+ * @throws IOException
+ */
+ public abstract TimelineEntity getApplicationEntity(
+ ApplicationId appId, String fields, Map filters)
+ throws IOException;
+
+ /**
+ * Gets application attempt entity.
+ * @param appAttemptId application attempt id
+ * @param fields Fields to be fetched. Defaults to INFO.
+ * @param filters Filters to be applied while fetching entities.
+ * @return entity associated with application attempt
+ * @throws IOException
+ */
+ public abstract TimelineEntity getApplicationAttemptEntity(
+ ApplicationAttemptId appAttemptId, String fields,
+ Map filters) throws IOException;
+
+ /**
+ * Gets application attempt entities.
+ * @param appId application id
+ * @param
[2/2] hadoop git commit: YARN-8303. YarnClient should contact TimelineReader for application/attempt/container report.
YARN-8303. YarnClient should contact TimelineReader for
application/attempt/container report.
(cherry picked from commit ee3355be3cd1cd956296a1b79b264177ff139ce6)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/13e3670e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/13e3670e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/13e3670e
Branch: refs/heads/branch-3.2
Commit: 13e3670e7f67806ec69b0e9f0f55022452ee06fe
Parents: da05328
Author: Rohith Sharma K S
Authored: Fri Nov 16 17:51:40 2018 +0530
Committer: Rohith Sharma K S
Committed: Fri Nov 16 18:37:20 2018 +0530
--
.../hadoop/yarn/client/api/AHSClient.java | 10 +-
.../yarn/client/api/impl/AHSv2ClientImpl.java | 149 ++
.../yarn/client/api/impl/YarnClientImpl.java| 194 +---
.../apache/hadoop/yarn/client/cli/LogsCLI.java | 13 +-
.../client/api/impl/TestAHSv2ClientImpl.java| 240 ++
.../hadoop/yarn/client/cli/TestLogsCLI.java | 3 +
.../metrics/AppAttemptMetricsConstants.java | 69 +++
.../metrics/ApplicationMetricsConstants.java| 122 +
.../metrics/ContainerMetricsConstants.java | 83
.../yarn/server/metrics/package-info.java | 22 +
.../timeline/TimelineEntityV2Converter.java | 449 +++
.../metrics/AppAttemptMetricsConstants.java | 69 ---
.../metrics/ApplicationMetricsConstants.java| 122 -
.../metrics/ContainerMetricsConstants.java | 83
14 files changed, 1288 insertions(+), 340 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/13e3670e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
index b590a51..6da8e66 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
@@ -33,6 +33,7 @@ import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.api.records.ContainerId;
import org.apache.hadoop.yarn.api.records.ContainerReport;
import org.apache.hadoop.yarn.client.api.impl.AHSClientImpl;
+import org.apache.hadoop.yarn.client.api.impl.AHSv2ClientImpl;
import org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException;
import org.apache.hadoop.yarn.exceptions.ContainerNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
@@ -46,8 +47,13 @@ public abstract class AHSClient extends AbstractService {
*/
@Public
public static AHSClient createAHSClient() {
-AHSClient client = new AHSClientImpl();
-return client;
+return new AHSClientImpl();
+ }
+
+ @InterfaceStability.Evolving
+ @Public
+ public static AHSClient createAHSv2Client() {
+return new AHSv2ClientImpl();
}
@Private
http://git-wip-us.apache.org/repos/asf/hadoop/blob/13e3670e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
new file mode 100644
index 000..e797c28
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
@@ -0,0 +1,149 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
[2/2] hadoop git commit: YARN-8834 Provide Java client for fetching Yarn specific entities from TimelineReader. Contributed by Abhishek Modi
YARN-8834 Provide Java client for fetching Yarn specific entities from
TimelineReader. Contributed by Abhishek Modi
(cherry picked from commit a3edfddcf7822ea13bdf4858672eb82cea5e0b5f)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1dd2cf29
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1dd2cf29
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1dd2cf29
Branch: refs/heads/branch-3.1
Commit: 1dd2cf296d5be8375f3674902ca4c51b21dacfb6
Parents: e5e4dc4
Author: Vrushali C
Authored: Thu Oct 11 20:21:00 2018 -0700
Committer: Rohith Sharma K S
Committed: Fri Nov 16 18:38:11 2018 +0530
--
.../yarn/client/api/TimelineReaderClient.java | 120 ++
.../api/impl/TimelineReaderClientImpl.java | 239 +++
.../api/impl/TestTimelineReaderClientImpl.java | 157
3 files changed, 516 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1dd2cf29/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineReaderClient.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineReaderClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineReaderClient.java
new file mode 100644
index 000..f73c2d3
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineReaderClient.java
@@ -0,0 +1,120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.client.api;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.service.CompositeService;
+import org.apache.hadoop.yarn.api.records.ApplicationAttemptId;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ContainerId;
+import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity;
+import org.apache.hadoop.yarn.client.api.impl.TimelineReaderClientImpl;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A client library that can be used to get Timeline Entities associated with
+ * application, application attempt or containers. This client library needs to
+ * be used along with time line v.2 server version.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public abstract class TimelineReaderClient extends CompositeService {
+
+ /**
+ * Create a new instance of Timeline Reader Client.
+ */
+ @InterfaceAudience.Public
+ public static TimelineReaderClient createTimelineReaderClient() {
+return new TimelineReaderClientImpl();
+ }
+
+ @InterfaceAudience.Private
+ public TimelineReaderClient(String name) {
+super(name);
+ }
+
+ /**
+ * Gets application entity.
+ * @param appId application id
+ * @param fields Fields to be fetched. Defaults to INFO.
+ * @param filters Filters to be applied while fetching entities.
+ * @return entity of the application
+ * @throws IOException
+ */
+ public abstract TimelineEntity getApplicationEntity(
+ ApplicationId appId, String fields, Map filters)
+ throws IOException;
+
+ /**
+ * Gets application attempt entity.
+ * @param appAttemptId application attempt id
+ * @param fields Fields to be fetched. Defaults to INFO.
+ * @param filters Filters to be applied while fetching entities.
+ * @return entity associated with application attempt
+ * @throws IOException
+ */
+ public abstract TimelineEntity getApplicationAttemptEntity(
+ ApplicationAttemptId appAttemptId, String fields,
+ Map filters) throws IOException;
+
+ /**
+ * Gets application attempt entities.
+ * @param appId application id
+ * @param fields Fields to be fetched. Defaults to INFO.
+ * @param filters Filters to be
[1/2] hadoop git commit: YARN-8303. YarnClient should contact TimelineReader for application/attempt/container report.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 e5e4dc42a -> 095635d98
YARN-8303. YarnClient should contact TimelineReader for
application/attempt/container report.
(cherry picked from commit ee3355be3cd1cd956296a1b79b264177ff139ce6)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/095635d9
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/095635d9
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/095635d9
Branch: refs/heads/branch-3.1
Commit: 095635d984fbc6ac7bead789ae5d134ac4ede848
Parents: 1dd2cf2
Author: Rohith Sharma K S
Authored: Fri Nov 16 17:51:40 2018 +0530
Committer: Rohith Sharma K S
Committed: Fri Nov 16 18:38:11 2018 +0530
--
.../hadoop/yarn/client/api/AHSClient.java | 10 +-
.../yarn/client/api/impl/AHSv2ClientImpl.java | 149 ++
.../yarn/client/api/impl/YarnClientImpl.java| 194 +---
.../apache/hadoop/yarn/client/cli/LogsCLI.java | 13 +-
.../client/api/impl/TestAHSv2ClientImpl.java| 240 ++
.../hadoop/yarn/client/cli/TestLogsCLI.java | 3 +
.../metrics/AppAttemptMetricsConstants.java | 69 +++
.../metrics/ApplicationMetricsConstants.java| 122 +
.../metrics/ContainerMetricsConstants.java | 83
.../yarn/server/metrics/package-info.java | 22 +
.../timeline/TimelineEntityV2Converter.java | 449 +++
.../metrics/AppAttemptMetricsConstants.java | 69 ---
.../metrics/ApplicationMetricsConstants.java| 122 -
.../metrics/ContainerMetricsConstants.java | 83
14 files changed, 1288 insertions(+), 340 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/095635d9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
index b590a51..6da8e66 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
@@ -33,6 +33,7 @@ import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.api.records.ContainerId;
import org.apache.hadoop.yarn.api.records.ContainerReport;
import org.apache.hadoop.yarn.client.api.impl.AHSClientImpl;
+import org.apache.hadoop.yarn.client.api.impl.AHSv2ClientImpl;
import org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException;
import org.apache.hadoop.yarn.exceptions.ContainerNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
@@ -46,8 +47,13 @@ public abstract class AHSClient extends AbstractService {
*/
@Public
public static AHSClient createAHSClient() {
-AHSClient client = new AHSClientImpl();
-return client;
+return new AHSClientImpl();
+ }
+
+ @InterfaceStability.Evolving
+ @Public
+ public static AHSClient createAHSv2Client() {
+return new AHSv2ClientImpl();
}
@Private
http://git-wip-us.apache.org/repos/asf/hadoop/blob/095635d9/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
new file mode 100644
index 000..e797c28
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
@@ -0,0 +1,149 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing
hadoop git commit: YARN-8303. YarnClient should contact TimelineReader for application/attempt/container report.
Repository: hadoop
Updated Branches:
refs/heads/trunk 8571507ef -> ee3355be3
YARN-8303. YarnClient should contact TimelineReader for
application/attempt/container report.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ee3355be
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ee3355be
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ee3355be
Branch: refs/heads/trunk
Commit: ee3355be3cd1cd956296a1b79b264177ff139ce6
Parents: 8571507
Author: Rohith Sharma K S
Authored: Fri Nov 16 17:51:40 2018 +0530
Committer: Rohith Sharma K S
Committed: Fri Nov 16 17:58:19 2018 +0530
--
.../hadoop/yarn/client/api/AHSClient.java | 10 +-
.../yarn/client/api/impl/AHSv2ClientImpl.java | 149 ++
.../yarn/client/api/impl/YarnClientImpl.java| 194 +---
.../apache/hadoop/yarn/client/cli/LogsCLI.java | 13 +-
.../client/api/impl/TestAHSv2ClientImpl.java| 240 ++
.../hadoop/yarn/client/cli/TestLogsCLI.java | 3 +
.../metrics/AppAttemptMetricsConstants.java | 69 +++
.../metrics/ApplicationMetricsConstants.java| 122 +
.../metrics/ContainerMetricsConstants.java | 83
.../yarn/server/metrics/package-info.java | 22 +
.../timeline/TimelineEntityV2Converter.java | 449 +++
.../metrics/AppAttemptMetricsConstants.java | 69 ---
.../metrics/ApplicationMetricsConstants.java| 122 -
.../metrics/ContainerMetricsConstants.java | 83
14 files changed, 1288 insertions(+), 340 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ee3355be/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
index b590a51..6da8e66 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AHSClient.java
@@ -33,6 +33,7 @@ import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.api.records.ContainerId;
import org.apache.hadoop.yarn.api.records.ContainerReport;
import org.apache.hadoop.yarn.client.api.impl.AHSClientImpl;
+import org.apache.hadoop.yarn.client.api.impl.AHSv2ClientImpl;
import org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException;
import org.apache.hadoop.yarn.exceptions.ContainerNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
@@ -46,8 +47,13 @@ public abstract class AHSClient extends AbstractService {
*/
@Public
public static AHSClient createAHSClient() {
-AHSClient client = new AHSClientImpl();
-return client;
+return new AHSClientImpl();
+ }
+
+ @InterfaceStability.Evolving
+ @Public
+ public static AHSClient createAHSv2Client() {
+return new AHSv2ClientImpl();
}
@Private
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ee3355be/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
new file mode 100644
index 000..e797c28
--- /dev/null
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java
@@ -0,0 +1,149 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package
hadoop git commit: YARN-8006. Make Hbase-2 profile as default for YARN-7055 branch.
Repository: hadoop Updated Branches: refs/heads/YARN-7055 486b9a4a7 -> 0a32c126f YARN-8006. Make Hbase-2 profile as default for YARN-7055 branch. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0a32c126 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0a32c126 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0a32c126 Branch: refs/heads/YARN-7055 Commit: 0a32c126fcfeeaac129a968d59510c1d414e7628 Parents: 486b9a4 Author: Rohith Sharma K S Authored: Wed Oct 31 10:26:42 2018 +0530 Committer: Rohith Sharma K S Committed: Wed Oct 31 10:26:42 2018 +0530 -- hadoop-project/pom.xml | 6 +++--- .../hadoop-yarn-server-timelineservice-hbase-tests/pom.xml | 6 +++--- .../hadoop-yarn-server-timelineservice-hbase-server/pom.xml| 6 +++--- 3 files changed, 9 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/0a32c126/hadoop-project/pom.xml -- diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 2247109..530ec42 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -1951,7 +1951,8 @@ hbase1 - !hbase.profile + hbase.profile + 1.0 @@ -1977,8 +1978,7 @@ hbase2 - hbase.profile - 2.0 + !hbase.profile http://git-wip-us.apache.org/repos/asf/hadoop/blob/0a32c126/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml index f1ca3ea..b3c4af1 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml @@ -426,7 +426,8 @@ hbase1 - !hbase.profile + hbase.profile + 1.0 @@ -448,8 +449,7 @@ hbase2 - hbase.profile - 2.0 + !hbase.profile http://git-wip-us.apache.org/repos/asf/hadoop/blob/0a32c126/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/pom.xml -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/pom.xml index 816bf37..f3eabdc 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/pom.xml @@ -36,7 +36,8 @@ hbase1 - !hbase.profile + hbase.profile + 1.0 @@ -60,8 +61,7 @@ hbase2 - hbase.profile - 2.0 + !hbase.profile - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0.
Repository: hadoop Updated Branches: refs/heads/branch-3.2 8788489df -> 7920f7904 YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0. (cherry picked from commit 4ec4ec69711180d642c5b56cd3d3dbdf44d3c61f) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7920f790 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7920f790 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7920f790 Branch: refs/heads/branch-3.2 Commit: 7920f790491155fdb97e70ee7e6cd9041362ed43 Parents: 8788489 Author: Rohith Sharma K S Authored: Tue Oct 30 11:29:58 2018 +0530 Committer: Rohith Sharma K S Committed: Tue Oct 30 11:50:33 2018 +0530 -- .../hadoop-yarn-server-timelineservice-hbase-client/pom.xml | 8 .../pom.xml | 8 2 files changed, 16 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/7920f790/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml index e978936..048ca7d 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml @@ -160,6 +160,14 @@ org.mortbay.jetty jetty-sslengine + + org.eclipse.jetty + jetty-security + + + org.eclipse.jetty + jetty-http + http://git-wip-us.apache.org/repos/asf/hadoop/blob/7920f790/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml index 564e79c..ed2703f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml @@ -147,6 +147,14 @@ org.mortbay.jetty jetty-sslengine + + org.eclipse.jetty + jetty-security + + + org.eclipse.jetty + jetty-http + - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0.
Repository: hadoop Updated Branches: refs/heads/branch-3.1 399645ebc -> d9a494b1e YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0. (cherry picked from commit 4ec4ec69711180d642c5b56cd3d3dbdf44d3c61f) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d9a494b1 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d9a494b1 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d9a494b1 Branch: refs/heads/branch-3.1 Commit: d9a494b1e043e387ba85e7f6753246dfc4c641eb Parents: 399645e Author: Rohith Sharma K S Authored: Tue Oct 30 11:29:58 2018 +0530 Committer: Rohith Sharma K S Committed: Tue Oct 30 11:50:56 2018 +0530 -- .../hadoop-yarn-server-timelineservice-hbase-client/pom.xml | 8 .../pom.xml | 8 2 files changed, 16 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/d9a494b1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml index 2062eb6..cf1c765 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml @@ -165,6 +165,14 @@ org.mortbay.jetty jetty-sslengine + + org.eclipse.jetty + jetty-security + + + org.eclipse.jetty + jetty-http + http://git-wip-us.apache.org/repos/asf/hadoop/blob/d9a494b1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml index 1e58914..4fd5461 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml @@ -147,6 +147,14 @@ org.mortbay.jetty jetty-sslengine + + org.eclipse.jetty + jetty-security + + + org.eclipse.jetty + jetty-http + - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[2/3] hadoop git commit: YARN-8871. Document ATSv2 integrated LogWebService. Contributed by Suma Shivaprasad.
YARN-8871. Document ATSv2 integrated LogWebService. Contributed by Suma
Shivaprasad.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a283da21
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a283da21
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a283da21
Branch: refs/heads/trunk
Commit: a283da21670c812a391d4c9ee98ebef22fc93868
Parents: 4ec4ec6
Author: Rohith Sharma K S
Authored: Tue Oct 30 11:34:20 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Oct 30 11:34:20 2018 +0530
--
.../hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md | 5 +
1 file changed, 5 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a283da21/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
index 04948ce..2314f30 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
@@ -1570,3 +1570,8 @@ With this API, you can query set of available entity
types for a given app id. I
1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is
returned.
1. If flow context information cannot be retrieved or entity for the given
entity id cannot be found, HTTP 404 (Not Found) is returned.
1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server
Error) is returned.
+
+## Aggregated Log
Serving for Historical Apps
+
+ TimelineService v.2 supports serving aggregated logs of historical apps. To
enable this, configure "yarn.log.server.web-service.url" to "${yarn
.timeline-service.hostname}:8188/ws/v2/applicationlog"
+ in `yarn-site.xml`
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[3/3] hadoop git commit: YARN-7754. [Atsv2] Update document for running v1 and v2 TS. Contributed by Suma Shivaprasad.
YARN-7754. [Atsv2] Update document for running v1 and v2 TS. Contributed by Suma Shivaprasad. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/486b9a4a Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/486b9a4a Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/486b9a4a Branch: refs/heads/trunk Commit: 486b9a4a75f413aa542338b0d866c3b490381d93 Parents: a283da2 Author: Rohith Sharma K S Authored: Tue Oct 30 11:35:01 2018 +0530 Committer: Rohith Sharma K S Committed: Tue Oct 30 11:35:01 2018 +0530 -- .../src/site/markdown/TimelineServiceV2.md | 12 1 file changed, 12 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/486b9a4a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md index 2314f30..86faf6c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md @@ -333,6 +333,18 @@ that it can write data to the Apache HBase cluster you are using, or set ``` +To configure both Timeline Service 1.5 and v.2, add the following property + + ``` + + yarn.timeline-service.versions + 1.5f,2.0f + +``` + +If the above is not configured, then it defaults to the version set in `yarn.timeline-service.version` + + Running Timeline Service v.2 Restart the resource manager as well as the node managers to pick up the new configuration. The collectors start within the resource manager and the node managers in an embedded manner. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[1/3] hadoop git commit: YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0.
Repository: hadoop Updated Branches: refs/heads/trunk db7e63682 -> 486b9a4a7 YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4ec4ec69 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4ec4ec69 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4ec4ec69 Branch: refs/heads/trunk Commit: 4ec4ec69711180d642c5b56cd3d3dbdf44d3c61f Parents: db7e636 Author: Rohith Sharma K S Authored: Tue Oct 30 11:29:58 2018 +0530 Committer: Rohith Sharma K S Committed: Tue Oct 30 11:30:08 2018 +0530 -- .../hadoop-yarn-server-timelineservice-hbase-client/pom.xml | 8 .../pom.xml | 8 2 files changed, 16 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/4ec4ec69/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml index 86b2158..4225519 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client/pom.xml @@ -160,6 +160,14 @@ org.mortbay.jetty jetty-sslengine + + org.eclipse.jetty + jetty-security + + + org.eclipse.jetty + jetty-http + http://git-wip-us.apache.org/repos/asf/hadoop/blob/4ec4ec69/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml -- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml index 4fde40c..984cac9 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2/pom.xml @@ -147,6 +147,14 @@ org.mortbay.jetty jetty-sslengine + + org.eclipse.jetty + jetty-security + + + org.eclipse.jetty + jetty-http + - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.0 d8cd07d5f -> 6ff7056b5
YARN-8826. Fix lingering timeline collector after serviceStop in
TimelineCollectorManager. Contributed by Prabha Manepalli.
(cherry picked from commit 0b62983c5a9361eb832784f134f140f9926c9ec6)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/6ff7056b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/6ff7056b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/6ff7056b
Branch: refs/heads/branch-3.0
Commit: 6ff7056b58b0b8112a28db3c31e2d2b7f04553d4
Parents: d8cd07d
Author: Rohith Sharma K S
Authored: Tue Oct 23 12:58:41 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Oct 23 14:09:18 2018 +0530
--
.../server/timelineservice/collector/TimelineCollectorManager.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6ff7056b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
index 7909a2e..d59b2e0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
@@ -220,7 +220,7 @@ public class TimelineCollectorManager extends
CompositeService {
@Override
protected void serviceStop() throws Exception {
-if (collectors != null && collectors.size() > 1) {
+if (collectors != null && collectors.size() > 0) {
synchronized (collectors) {
for (TimelineCollector c : collectors.values()) {
c.serviceStop();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli.
Repository: hadoop
Updated Branches:
refs/heads/branch-2 56a27284a -> dbf36dbb8
YARN-8826. Fix lingering timeline collector after serviceStop in
TimelineCollectorManager. Contributed by Prabha Manepalli.
(cherry picked from commit 0b62983c5a9361eb832784f134f140f9926c9ec6)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/dbf36dbb
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/dbf36dbb
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/dbf36dbb
Branch: refs/heads/branch-2
Commit: dbf36dbb8dca8c953aaf26e3bbfb5ec5ce3358ea
Parents: 56a2728
Author: Rohith Sharma K S
Authored: Tue Oct 23 12:58:41 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Oct 23 14:10:24 2018 +0530
--
.../server/timelineservice/collector/TimelineCollectorManager.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/dbf36dbb/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
index c64a3ad..9727850 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
@@ -221,7 +221,7 @@ public class TimelineCollectorManager extends
CompositeService {
@Override
protected void serviceStop() throws Exception {
-if (collectors != null && collectors.size() > 1) {
+if (collectors != null && collectors.size() > 0) {
synchronized (collectors) {
for (TimelineCollector c : collectors.values()) {
c.serviceStop();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 0cb184d6e -> 3e3b08885
YARN-8826. Fix lingering timeline collector after serviceStop in
TimelineCollectorManager. Contributed by Prabha Manepalli.
(cherry picked from commit 0b62983c5a9361eb832784f134f140f9926c9ec6)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3e3b0888
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3e3b0888
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3e3b0888
Branch: refs/heads/branch-3.1
Commit: 3e3b088856b611ce47e360e1b01b7cd23c65cd67
Parents: 0cb184d
Author: Rohith Sharma K S
Authored: Tue Oct 23 12:58:41 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Oct 23 14:08:06 2018 +0530
--
.../server/timelineservice/collector/TimelineCollectorManager.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3e3b0888/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
index 7909a2e..d59b2e0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
@@ -220,7 +220,7 @@ public class TimelineCollectorManager extends
CompositeService {
@Override
protected void serviceStop() throws Exception {
-if (collectors != null && collectors.size() > 1) {
+if (collectors != null && collectors.size() > 0) {
synchronized (collectors) {
for (TimelineCollector c : collectors.values()) {
c.serviceStop();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.2 30998fea2 -> 660fff313
YARN-8826. Fix lingering timeline collector after serviceStop in
TimelineCollectorManager. Contributed by Prabha Manepalli.
(cherry picked from commit 0b62983c5a9361eb832784f134f140f9926c9ec6)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/660fff31
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/660fff31
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/660fff31
Branch: refs/heads/branch-3.2
Commit: 660fff3138fce9a877fd0298f394c075cbcdcaaf
Parents: 30998fe
Author: Rohith Sharma K S
Authored: Tue Oct 23 12:58:41 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Oct 23 14:06:13 2018 +0530
--
.../server/timelineservice/collector/TimelineCollectorManager.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/660fff31/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
index 7909a2e..d59b2e0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
@@ -220,7 +220,7 @@ public class TimelineCollectorManager extends
CompositeService {
@Override
protected void serviceStop() throws Exception {
-if (collectors != null && collectors.size() > 1) {
+if (collectors != null && collectors.size() > 0) {
synchronized (collectors) {
for (TimelineCollector c : collectors.values()) {
c.serviceStop();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli.
Repository: hadoop
Updated Branches:
refs/heads/trunk 0efddd85f -> 0b62983c5
YARN-8826. Fix lingering timeline collector after serviceStop in
TimelineCollectorManager. Contributed by Prabha Manepalli.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0b62983c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0b62983c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0b62983c
Branch: refs/heads/trunk
Commit: 0b62983c5a9361eb832784f134f140f9926c9ec6
Parents: 0efddd8
Author: Rohith Sharma K S
Authored: Tue Oct 23 12:58:41 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Oct 23 12:58:41 2018 +0530
--
.../server/timelineservice/collector/TimelineCollectorManager.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/0b62983c/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
index 7909a2e..d59b2e0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager.java
@@ -220,7 +220,7 @@ public class TimelineCollectorManager extends
CompositeService {
@Override
protected void serviceStop() throws Exception {
-if (collectors != null && collectors.size() > 1) {
+if (collectors != null && collectors.size() > 0) {
synchronized (collectors) {
for (TimelineCollector c : collectors.values()) {
c.serviceStop();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8270 Adding JMX Metrics for Timeline Collector and Reader. Contributed by Sushil Ks.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 a56a345e0 -> 5b72aa04e
YARN-8270 Adding JMX Metrics for Timeline Collector and Reader. Contributed by
Sushil Ks.
(cherry picked from commit 90e2e493b3dc8be54f655b957b98a4bc0e003684)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/5b72aa04
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/5b72aa04
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/5b72aa04
Branch: refs/heads/branch-3.1
Commit: 5b72aa04e104242d1761abf56822fb38e9915def
Parents: a56a345
Author: Vrushali C
Authored: Thu Sep 27 15:53:39 2018 -0700
Committer: Rohith Sharma K S
Committed: Fri Sep 28 10:31:38 2018 +0530
--
.../collector/TimelineCollectorWebService.java | 16 +-
.../PerNodeAggTimelineCollectorMetrics.java | 117 +
.../metrics/TimelineReaderMetrics.java | 113 +
.../timelineservice/metrics/package-info.java | 28
.../reader/TimelineReaderWebServices.java | 166 +--
.../TestPerNodeAggTimelineCollectorMetrics.java | 56 +++
.../reader/TestTimelineReaderMetrics.java | 56 +++
7 files changed, 500 insertions(+), 52 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/5b72aa04/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
index 4e51fca..150249b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorWebService.java
@@ -41,6 +41,7 @@ import
org.apache.hadoop.classification.InterfaceAudience.Public;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
import org.apache.hadoop.http.JettyUtils;
import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.Time;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import
org.apache.hadoop.yarn.api.records.timelineservice.ApplicationAttemptEntity;
import org.apache.hadoop.yarn.api.records.timelineservice.ApplicationEntity;
@@ -53,6 +54,7 @@ import
org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntities;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntity;
import org.apache.hadoop.yarn.api.records.timelineservice.TimelineEntityType;
import org.apache.hadoop.yarn.api.records.timelineservice.UserEntity;
+import
org.apache.hadoop.yarn.server.timelineservice.metrics.PerNodeAggTimelineCollectorMetrics;
import org.apache.hadoop.yarn.webapp.ForbiddenException;
import org.apache.hadoop.yarn.webapp.NotFoundException;
@@ -77,6 +79,8 @@ public class TimelineCollectorWebService {
LoggerFactory.getLogger(TimelineCollectorWebService.class);
private @Context ServletContext context;
+ private static final PerNodeAggTimelineCollectorMetrics METRICS =
+ PerNodeAggTimelineCollectorMetrics.getInstance();
/**
* Gives information about timeline collector.
@@ -151,12 +155,15 @@ public class TimelineCollectorWebService {
TimelineEntities entities) {
init(res);
UserGroupInformation callerUgi = getUser(req);
+boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
if (callerUgi == null) {
String msg = "The owner of the posted timeline entities is not set";
LOG.error(msg);
throw new ForbiddenException(msg);
}
+long startTime = Time.monotonicNow();
+boolean succeeded = false;
try {
ApplicationId appID = parseApplicationId(appId);
if (appID == null) {
@@ -171,7 +178,6 @@ public class TimelineCollectorWebService {
throw new NotFoundException("Application: "+ appId + " is not found");
}
- boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
if (isAsync) {
collector.putEntitiesAsync(processTimelineEntities(entities, appId,
Boolean.valueOf(isSubAppEntities)), callerUgi);
@@ -180,6 +186,7 @@ public class TimelineCollectorWebService {
hadoop git commit: YARN-8824. App Nodelabel missed after RM restart for finished apps. Contributed by Bibin A Chundatt.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 7dbdb7560 -> d473152e6
YARN-8824. App Nodelabel missed after RM restart for finished apps. Contributed
by Bibin A Chundatt.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d473152e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d473152e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d473152e
Branch: refs/heads/branch-3.1
Commit: d473152e6aa69b63b641f122350a0343bc6d78e7
Parents: 7dbdb75
Author: Rohith Sharma K S
Authored: Wed Sep 26 20:36:34 2018 +0530
Committer: Rohith Sharma K S
Committed: Wed Sep 26 20:36:34 2018 +0530
--
.../resourcemanager/recovery/RMStateStore.java | 1 +
.../resourcemanager/MockMemoryRMStateStore.java | 1 +
.../yarn/server/resourcemanager/MockRM.java | 26 ---
.../server/resourcemanager/TestRMRestart.java | 49
4 files changed, 70 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d473152e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
index 1265b56..0fc98df 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
@@ -306,6 +306,7 @@ public abstract class RMStateStore extends AbstractService {
context.setApplicationTags(srcCtx.getApplicationTags());
context.setApplicationType(srcCtx.getApplicationType());
context.setUnmanagedAM(srcCtx.getUnmanagedAM());
+ context.setNodeLabelExpression(srcCtx.getNodeLabelExpression());
ContainerLaunchContextPBImpl amContainerSpec =
new ContainerLaunchContextPBImpl();
amContainerSpec.setApplicationACLs(
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d473152e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockMemoryRMStateStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockMemoryRMStateStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockMemoryRMStateStore.java
index 698f1c6..ef9606f 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockMemoryRMStateStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockMemoryRMStateStore.java
@@ -105,6 +105,7 @@ public class MockMemoryRMStateStore extends
MemoryRMStateStore {
oldAppSubCtxt.getAMContainerResourceRequests());
context.setLogAggregationContext(oldAppSubCtxt.getLogAggregationContext());
context.setApplicationType(oldAppSubCtxt.getApplicationType());
+context.setNodeLabelExpression(oldAppSubCtxt.getNodeLabelExpression());
this.appSubCtxtCopy.put(oldAppSubCtxt.getApplicationId(), context);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d473152e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
index 074dcfd..ef7d6cf 100644
---
hadoop git commit: YARN-8824. App Nodelabel missed after RM restart for finished apps. Contributed by Bibin A Chundatt.
Repository: hadoop
Updated Branches:
refs/heads/trunk 3f6195045 -> e5287a4fe
YARN-8824. App Nodelabel missed after RM restart for finished apps. Contributed
by Bibin A Chundatt.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e5287a4f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e5287a4f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e5287a4f
Branch: refs/heads/trunk
Commit: e5287a4fe0bb03d929f066fc50eb0e7bd74bb759
Parents: 3f61950
Author: Rohith Sharma K S
Authored: Wed Sep 26 12:30:26 2018 +0530
Committer: Rohith Sharma K S
Committed: Wed Sep 26 12:30:26 2018 +0530
--
.../resourcemanager/recovery/RMStateStore.java | 1 +
.../resourcemanager/MockMemoryRMStateStore.java | 1 +
.../yarn/server/resourcemanager/MockRM.java | 31 +
.../server/resourcemanager/TestRMRestart.java | 49
4 files changed, 82 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e5287a4f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
index a0b10b1..ccd6fc9 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
@@ -307,6 +307,7 @@ public abstract class RMStateStore extends AbstractService {
context.setApplicationTags(srcCtx.getApplicationTags());
context.setApplicationType(srcCtx.getApplicationType());
context.setUnmanagedAM(srcCtx.getUnmanagedAM());
+ context.setNodeLabelExpression(srcCtx.getNodeLabelExpression());
ContainerLaunchContextPBImpl amContainerSpec =
new ContainerLaunchContextPBImpl();
amContainerSpec.setApplicationACLs(
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e5287a4f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockMemoryRMStateStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockMemoryRMStateStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockMemoryRMStateStore.java
index 698f1c6..ef9606f 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockMemoryRMStateStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockMemoryRMStateStore.java
@@ -105,6 +105,7 @@ public class MockMemoryRMStateStore extends
MemoryRMStateStore {
oldAppSubCtxt.getAMContainerResourceRequests());
context.setLogAggregationContext(oldAppSubCtxt.getLogAggregationContext());
context.setApplicationType(oldAppSubCtxt.getApplicationType());
+context.setNodeLabelExpression(oldAppSubCtxt.getNodeLabelExpression());
this.appSubCtxtCopy.put(oldAppSubCtxt.getApplicationId(), context);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e5287a4f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
index 6a024f5..df86f28 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
+++
hadoop git commit: YARN-8815. RM fails to recover finished unmanaged AM. Contributed by Bibin A Chundatt.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 daa14bf06 -> 47306cc2d
YARN-8815. RM fails to recover finished unmanaged AM. Contributed by Bibin A
Chundatt.
(cherry picked from commit 50bc7746d7593bf6adc92fc9807a99cf53b4aef3)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/47306cc2
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/47306cc2
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/47306cc2
Branch: refs/heads/branch-3.1
Commit: 47306cc2db55bec1cf503143b0c846577e3f016c
Parents: daa14bf
Author: Rohith Sharma K S
Authored: Tue Sep 25 11:31:14 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Sep 25 11:40:09 2018 +0530
--
.../resourcemanager/recovery/RMStateStore.java | 1 +
.../yarn/server/resourcemanager/MockRM.java | 9 +
.../server/resourcemanager/TestRMRestart.java| 19 +++
3 files changed, 29 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/47306cc2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
index b4dd378..1265b56 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
@@ -305,6 +305,7 @@ public abstract class RMStateStore extends AbstractService {
context.setPriority(srcCtx.getPriority());
context.setApplicationTags(srcCtx.getApplicationTags());
context.setApplicationType(srcCtx.getApplicationType());
+ context.setUnmanagedAM(srcCtx.getUnmanagedAM());
ContainerLaunchContextPBImpl amContainerSpec =
new ContainerLaunchContextPBImpl();
amContainerSpec.setApplicationACLs(
http://git-wip-us.apache.org/repos/asf/hadoop/blob/47306cc2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
index eb4c626..074dcfd 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
@@ -586,6 +586,15 @@ public class MockRM extends ResourceManager {
true, false, false, null, 0, null, true, null);
}
+ public RMApp submitApp(Resource resource, String name, String user,
+ Map acls, boolean unManaged, String queue)
+ throws Exception {
+return submitApp(resource, name, user, acls, unManaged, queue,
+super.getConfig().getInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS,
+YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS), null, null,
true,
+false, false, null, 0, null, true, null);
+ }
+
public RMApp submitApp(int masterMemory, String name, String user,
Map acls, String queue,
boolean waitForAccepted) throws Exception {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/47306cc2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
index
hadoop git commit: YARN-8815. RM fails to recover finished unmanaged AM. Contributed by Bibin A Chundatt.
Repository: hadoop
Updated Branches:
refs/heads/trunk d0bc6a27f -> 50bc7746d
YARN-8815. RM fails to recover finished unmanaged AM. Contributed by Bibin A
Chundatt.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/50bc7746
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/50bc7746
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/50bc7746
Branch: refs/heads/trunk
Commit: 50bc7746d7593bf6adc92fc9807a99cf53b4aef3
Parents: d0bc6a2
Author: Rohith Sharma K S
Authored: Tue Sep 25 11:31:14 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Sep 25 11:31:14 2018 +0530
--
.../resourcemanager/recovery/RMStateStore.java | 1 +
.../yarn/server/resourcemanager/MockRM.java | 9 +
.../server/resourcemanager/TestRMRestart.java| 19 +++
3 files changed, 29 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/50bc7746/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
index 242b5d0..a0b10b1 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java
@@ -306,6 +306,7 @@ public abstract class RMStateStore extends AbstractService {
context.setPriority(srcCtx.getPriority());
context.setApplicationTags(srcCtx.getApplicationTags());
context.setApplicationType(srcCtx.getApplicationType());
+ context.setUnmanagedAM(srcCtx.getUnmanagedAM());
ContainerLaunchContextPBImpl amContainerSpec =
new ContainerLaunchContextPBImpl();
amContainerSpec.setApplicationACLs(
http://git-wip-us.apache.org/repos/asf/hadoop/blob/50bc7746/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
index 2ad4391..6a024f5 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
@@ -599,6 +599,15 @@ public class MockRM extends ResourceManager {
true, false, false, null, 0, null, true, null);
}
+ public RMApp submitApp(Resource resource, String name, String user,
+ Map acls, boolean unManaged, String queue)
+ throws Exception {
+return submitApp(resource, name, user, acls, unManaged, queue,
+super.getConfig().getInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS,
+YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS), null, null,
true,
+false, false, null, 0, null, true, null);
+ }
+
public RMApp submitApp(int masterMemory, String name, String user,
Map acls, String queue,
boolean waitForAccepted) throws Exception {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/50bc7746/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
index 9aa5c53..a8b8d89 100644
---
hadoop git commit: YARN-8129. Improve error message for invalid value in fields attribute. Contributed by Abhishek Modi.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.0 53ce37ad2 -> 7278566f2
YARN-8129. Improve error message for invalid value in fields attribute.
Contributed by Abhishek Modi.
(cherry picked from commit d3fef7a5c5b83d27e87b5e49928254a7d1b935e5)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7278566f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7278566f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7278566f
Branch: refs/heads/branch-3.0
Commit: 7278566f27391c308b8f81e859487c8c6634e1da
Parents: 53ce37a
Author: Rohith Sharma K S
Authored: Tue Aug 21 11:58:07 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Aug 21 12:11:32 2018 +0530
--
.../timelineservice/reader/TimelineReaderWebServicesUtils.java | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7278566f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
index f83c1ac..ae19b21 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
@@ -214,7 +214,11 @@ public final class TimelineReaderWebServicesUtils {
String[] strs = str.split(delimiter);
EnumSet fieldList = EnumSet.noneOf(Field.class);
for (String s : strs) {
- fieldList.add(Field.valueOf(s.trim().toUpperCase()));
+ try {
+fieldList.add(Field.valueOf(s.trim().toUpperCase()));
+ } catch (IllegalArgumentException e) {
+throw new IllegalArgumentException(s + " is not a valid field.");
+ }
}
return fieldList;
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8129. Improve error message for invalid value in fields attribute. Contributed by Abhishek Modi.
Repository: hadoop
Updated Branches:
refs/heads/branch-2 18ebe1830 -> 1a53aab4d
YARN-8129. Improve error message for invalid value in fields attribute.
Contributed by Abhishek Modi.
(cherry picked from commit d3fef7a5c5b83d27e87b5e49928254a7d1b935e5)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1a53aab4
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1a53aab4
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1a53aab4
Branch: refs/heads/branch-2
Commit: 1a53aab4d634e6858769fd78472b078f2493d485
Parents: 18ebe18
Author: Rohith Sharma K S
Authored: Tue Aug 21 11:58:07 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Aug 21 12:11:31 2018 +0530
--
.../timelineservice/reader/TimelineReaderWebServicesUtils.java | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1a53aab4/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
index f83c1ac..ae19b21 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
@@ -214,7 +214,11 @@ public final class TimelineReaderWebServicesUtils {
String[] strs = str.split(delimiter);
EnumSet fieldList = EnumSet.noneOf(Field.class);
for (String s : strs) {
- fieldList.add(Field.valueOf(s.trim().toUpperCase()));
+ try {
+fieldList.add(Field.valueOf(s.trim().toUpperCase()));
+ } catch (IllegalArgumentException e) {
+throw new IllegalArgumentException(s + " is not a valid field.");
+ }
}
return fieldList;
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8129. Improve error message for invalid value in fields attribute. Contributed by Abhishek Modi.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 8df2eb811 -> 63d521433
YARN-8129. Improve error message for invalid value in fields attribute.
Contributed by Abhishek Modi.
(cherry picked from commit d3fef7a5c5b83d27e87b5e49928254a7d1b935e5)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/63d52143
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/63d52143
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/63d52143
Branch: refs/heads/branch-3.1
Commit: 63d52143323015f5672a04d793470a8595d0bc0c
Parents: 8df2eb8
Author: Rohith Sharma K S
Authored: Tue Aug 21 11:58:07 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Aug 21 12:11:12 2018 +0530
--
.../timelineservice/reader/TimelineReaderWebServicesUtils.java | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/63d52143/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
index f83c1ac..ae19b21 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
@@ -214,7 +214,11 @@ public final class TimelineReaderWebServicesUtils {
String[] strs = str.split(delimiter);
EnumSet fieldList = EnumSet.noneOf(Field.class);
for (String s : strs) {
- fieldList.add(Field.valueOf(s.trim().toUpperCase()));
+ try {
+fieldList.add(Field.valueOf(s.trim().toUpperCase()));
+ } catch (IllegalArgumentException e) {
+throw new IllegalArgumentException(s + " is not a valid field.");
+ }
}
return fieldList;
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8129. Improve error message for invalid value in fields attribute. Contributed by Abhishek Modi.
Repository: hadoop
Updated Branches:
refs/heads/trunk 770d9d9bb -> d3fef7a5c
YARN-8129. Improve error message for invalid value in fields attribute.
Contributed by Abhishek Modi.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d3fef7a5
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d3fef7a5
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d3fef7a5
Branch: refs/heads/trunk
Commit: d3fef7a5c5b83d27e87b5e49928254a7d1b935e5
Parents: 770d9d9
Author: Rohith Sharma K S
Authored: Tue Aug 21 11:58:07 2018 +0530
Committer: Rohith Sharma K S
Committed: Tue Aug 21 11:58:07 2018 +0530
--
.../timelineservice/reader/TimelineReaderWebServicesUtils.java | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d3fef7a5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
index efaecd2..63529a4 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServicesUtils.java
@@ -214,7 +214,11 @@ public final class TimelineReaderWebServicesUtils {
String[] strs = str.split(delimiter);
EnumSet fieldList = EnumSet.noneOf(Field.class);
for (String s : strs) {
- fieldList.add(Field.valueOf(s.trim().toUpperCase()));
+ try {
+fieldList.add(Field.valueOf(s.trim().toUpperCase()));
+ } catch (IllegalArgumentException e) {
+throw new IllegalArgumentException(s + " is not a valid field.");
+ }
}
return fieldList;
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[2/2] hadoop git commit: YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked. Contributed by Wangda Tan.
YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM
ContainerManager dispatcher get blocked. Contributed by Wangda Tan.
(cherry picked from commit 4aacbfff605262aaf3dbd926258afcadc86c72c0)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/675aa2bb
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/675aa2bb
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/675aa2bb
Branch: refs/heads/branch-3.0
Commit: 675aa2bbc05e2f900403f9b862d38ba0cf77c208
Parents: 8118b14
Author: Rohith Sharma K S
Authored: Sat Aug 18 10:26:55 2018 +0530
Committer: Rohith Sharma K S
Committed: Sat Aug 18 11:06:14 2018 +0530
--
.../applicationsmanager/TestAMLaunchFailure.java | 2 +-
.../applicationsmanager/TestSchedulerNegotiator.java | 2 +-
.../TestTimelineServiceClientIntegration.java | 3 ++-
.../security/TestTimelineAuthFilterForV2.java | 2 +-
.../collector/PerNodeTimelineCollectorsAuxService.java | 13 +
5 files changed, 14 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/675aa2bb/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
index c0009dd..ad39099 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
@@ -90,7 +90,7 @@ public class TestAMLaunchFailure {
//}
//
//@Override
-//public void addApplication(ApplicationId applicationId,
+//public void addApplicationIfAbsent(ApplicationId applicationId,
//ApplicationMaster master, String user, String queue, Priority
priority
//, ApplicationStore appStore)
//throws IOException {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/675aa2bb/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
index 7d06e55..fedbf2b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
@@ -67,7 +67,7 @@ public class TestSchedulerNegotiator {
// return null;
//}
//@Override
-//public void addApplication(ApplicationId applicationId,
+//public void addApplicationIfAbsent(ApplicationId applicationId,
//ApplicationMaster master, String user, String queue, Priority
priority,
//ApplicationStore store)
//throws IOException {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/675aa2bb/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java
hadoop git commit: YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked. Contributed by Wangda Tan.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 7556b09e9 -> a3d4a25bb
YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM
ContainerManager dispatcher get blocked. Contributed by Wangda Tan.
(cherry picked from commit 4aacbfff605262aaf3dbd926258afcadc86c72c0)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a3d4a25b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a3d4a25b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a3d4a25b
Branch: refs/heads/branch-3.1
Commit: a3d4a25bbfe5e41393f790e77b8e457f13c8424d
Parents: 7556b09
Author: Rohith Sharma K S
Authored: Sat Aug 18 10:26:55 2018 +0530
Committer: Rohith Sharma K S
Committed: Sat Aug 18 11:04:09 2018 +0530
--
.../applicationsmanager/TestAMLaunchFailure.java | 2 +-
.../applicationsmanager/TestSchedulerNegotiator.java | 2 +-
.../TestTimelineServiceClientIntegration.java | 3 ++-
.../security/TestTimelineAuthFilterForV2.java | 2 +-
.../collector/PerNodeTimelineCollectorsAuxService.java | 13 +
5 files changed, 14 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a3d4a25b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
index c0009dd..ad39099 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
@@ -90,7 +90,7 @@ public class TestAMLaunchFailure {
//}
//
//@Override
-//public void addApplication(ApplicationId applicationId,
+//public void addApplicationIfAbsent(ApplicationId applicationId,
//ApplicationMaster master, String user, String queue, Priority
priority
//, ApplicationStore appStore)
//throws IOException {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a3d4a25b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
index 7d06e55..fedbf2b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
@@ -67,7 +67,7 @@ public class TestSchedulerNegotiator {
// return null;
//}
//@Override
-//public void addApplication(ApplicationId applicationId,
+//public void addApplicationIfAbsent(ApplicationId applicationId,
//ApplicationMaster master, String user, String queue, Priority
priority,
//ApplicationStore store)
//throws IOException {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a3d4a25b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java
[1/2] hadoop git commit: YARN-7835. Race condition in NM while publishing events if second attempt is launched on the same node. (Rohith Sharma K S via Haibo Chen)
Repository: hadoop
Updated Branches:
refs/heads/branch-3.0 a41f18098 -> 675aa2bbc
YARN-7835. Race condition in NM while publishing events if second attempt is
launched on the same node. (Rohith Sharma K S via Haibo Chen)
(cherry picked from commit d1274c3b71549cb000868500c293cafd880b3713)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8118b14d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8118b14d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8118b14d
Branch: refs/heads/branch-3.0
Commit: 8118b14db84f8a4b5c955894b65f2f8d22be5254
Parents: a41f180
Author: Haibo Chen
Authored: Wed Feb 28 21:06:42 2018 -0800
Committer: Rohith Sharma K S
Committed: Sat Aug 18 11:06:02 2018 +0530
--
.../PerNodeTimelineCollectorsAuxService.java| 51 +--
...TestPerNodeTimelineCollectorsAuxService.java | 93
2 files changed, 120 insertions(+), 24 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8118b14d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/PerNodeTimelineCollectorsAuxService.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/PerNodeTimelineCollectorsAuxService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/PerNodeTimelineCollectorsAuxService.java
index 66f9aab..c15f99d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/PerNodeTimelineCollectorsAuxService.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/collector/PerNodeTimelineCollectorsAuxService.java
@@ -19,7 +19,12 @@
package org.apache.hadoop.yarn.server.timelineservice.collector;
import java.nio.ByteBuffer;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
@@ -31,6 +36,7 @@ import org.apache.hadoop.util.ShutdownHookManager;
import org.apache.hadoop.util.StringUtils;
import org.apache.hadoop.yarn.YarnUncaughtExceptionHandler;
import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.api.records.ContainerId;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.server.api.ApplicationInitializationContext;
@@ -59,6 +65,8 @@ public class PerNodeTimelineCollectorsAuxService extends
AuxiliaryService {
private final NodeTimelineCollectorManager collectorManager;
private long collectorLingerPeriod;
private ScheduledExecutorService scheduler;
+ private Map> appIdToContainerId =
+ new ConcurrentHashMap<>();
public PerNodeTimelineCollectorsAuxService() {
this(new NodeTimelineCollectorManager(true));
@@ -148,7 +156,15 @@ public class PerNodeTimelineCollectorsAuxService extends
AuxiliaryService {
if (context.getContainerType() == ContainerType.APPLICATION_MASTER) {
ApplicationId appId = context.getContainerId().
getApplicationAttemptId().getApplicationId();
- addApplication(appId, context.getUser());
+ synchronized (appIdToContainerId) {
+Set masterContainers = appIdToContainerId.get(appId);
+if (masterContainers == null) {
+ masterContainers = new HashSet<>();
+ appIdToContainerId.put(appId, masterContainers);
+}
+masterContainers.add(context.getContainerId());
+addApplication(appId, context.getUser());
+ }
}
}
@@ -162,17 +178,36 @@ public class PerNodeTimelineCollectorsAuxService extends
AuxiliaryService {
// intercept the event of the AM container being stopped and remove the app
// level collector service
if (context.getContainerType() == ContainerType.APPLICATION_MASTER) {
- final ApplicationId appId =
-
context.getContainerId().getApplicationAttemptId().getApplicationId();
- scheduler.schedule(new Runnable() {
-public void run() {
- removeApplication(appId);
-}
- }, collectorLingerPeriod, TimeUnit.MILLISECONDS);
+ final ContainerId containerId = context.getContainerId();
+
hadoop git commit: YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked. Contributed by Wangda Tan.
Repository: hadoop
Updated Branches:
refs/heads/branch-2 805647287 -> c68d1d49c
YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM
ContainerManager dispatcher get blocked. Contributed by Wangda Tan.
(cherry picked from commit 4aacbfff605262aaf3dbd926258afcadc86c72c0)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c68d1d49
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c68d1d49
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c68d1d49
Branch: refs/heads/branch-2
Commit: c68d1d49ca05eb9da9fd31b93c8c6ebd7717cce3
Parents: 8056472
Author: Rohith Sharma K S
Authored: Sat Aug 18 10:26:55 2018 +0530
Committer: Rohith Sharma K S
Committed: Sat Aug 18 11:04:20 2018 +0530
--
.../applicationsmanager/TestAMLaunchFailure.java | 2 +-
.../applicationsmanager/TestSchedulerNegotiator.java | 2 +-
.../TestTimelineServiceClientIntegration.java | 3 ++-
.../security/TestTimelineAuthFilterForV2.java | 2 +-
.../collector/PerNodeTimelineCollectorsAuxService.java | 13 +
5 files changed, 14 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c68d1d49/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
index c0009dd..ad39099 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
@@ -90,7 +90,7 @@ public class TestAMLaunchFailure {
//}
//
//@Override
-//public void addApplication(ApplicationId applicationId,
+//public void addApplicationIfAbsent(ApplicationId applicationId,
//ApplicationMaster master, String user, String queue, Priority
priority
//, ApplicationStore appStore)
//throws IOException {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c68d1d49/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
index 7d06e55..fedbf2b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
@@ -67,7 +67,7 @@ public class TestSchedulerNegotiator {
// return null;
//}
//@Override
-//public void addApplication(ApplicationId applicationId,
+//public void addApplicationIfAbsent(ApplicationId applicationId,
//ApplicationMaster master, String user, String queue, Priority
priority,
//ApplicationStore store)
//throws IOException {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c68d1d49/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java
hadoop git commit: YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked. Contributed by Wangda Tan.
Repository: hadoop
Updated Branches:
refs/heads/trunk 79c97f6a0 -> 4aacbfff6
YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM
ContainerManager dispatcher get blocked. Contributed by Wangda Tan.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4aacbfff
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4aacbfff
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4aacbfff
Branch: refs/heads/trunk
Commit: 4aacbfff605262aaf3dbd926258afcadc86c72c0
Parents: 79c97f6
Author: Rohith Sharma K S
Authored: Sat Aug 18 10:26:55 2018 +0530
Committer: Rohith Sharma K S
Committed: Sat Aug 18 10:26:55 2018 +0530
--
.../applicationsmanager/TestAMLaunchFailure.java | 2 +-
.../applicationsmanager/TestSchedulerNegotiator.java | 2 +-
.../TestTimelineServiceClientIntegration.java | 3 ++-
.../security/TestTimelineAuthFilterForV2.java | 2 +-
.../collector/PerNodeTimelineCollectorsAuxService.java | 13 +
5 files changed, 14 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/4aacbfff/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
index c0009dd..ad39099 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java
@@ -90,7 +90,7 @@ public class TestAMLaunchFailure {
//}
//
//@Override
-//public void addApplication(ApplicationId applicationId,
+//public void addApplicationIfAbsent(ApplicationId applicationId,
//ApplicationMaster master, String user, String queue, Priority
priority
//, ApplicationStore appStore)
//throws IOException {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/4aacbfff/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
index 7d06e55..fedbf2b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java
@@ -67,7 +67,7 @@ public class TestSchedulerNegotiator {
// return null;
//}
//@Override
-//public void addApplication(ApplicationId applicationId,
+//public void addApplicationIfAbsent(ApplicationId applicationId,
//ApplicationMaster master, String user, String queue, Priority
priority,
//ApplicationStore store)
//throws IOException {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/4aacbfff/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java
hadoop git commit: YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration. Contributed by Prabha Manepalli.
Repository: hadoop
Updated Branches:
refs/heads/branch-2.9 055e11fcc -> 50ba2272e
YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration.
Contributed by Prabha Manepalli.
(cherry picked from commit 1697a0230696e1ed6d9c19471463b44a6d791dfa)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/50ba2272
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/50ba2272
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/50ba2272
Branch: refs/heads/branch-2.9
Commit: 50ba2272e8d93f86b28552615269ed91a46be336
Parents: 055e11f
Author: Rohith Sharma K S
Authored: Fri Aug 17 11:11:56 2018 +0530
Committer: Rohith Sharma K S
Committed: Fri Aug 17 11:27:18 2018 +0530
--
.../main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/50ba2272/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 001d02e..2a8185e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1036,7 +1036,7 @@ public class YarnConfiguration extends Configuration {
NM_PREFIX + "collector-service.address";
public static final int DEFAULT_NM_COLLECTOR_SERVICE_PORT = 8048;
public static final String DEFAULT_NM_COLLECTOR_SERVICE_ADDRESS =
- "0.0.0.0:" + DEFAULT_NM_LOCALIZER_PORT;
+ "0.0.0.0:" + DEFAULT_NM_COLLECTOR_SERVICE_PORT;
/** Interval in between cache cleanups.*/
public static final String NM_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS =
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration. Contributed by Prabha Manepalli.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.0 27508086d -> 90bf2d3b5
YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration.
Contributed by Prabha Manepalli.
(cherry picked from commit 1697a0230696e1ed6d9c19471463b44a6d791dfa)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/90bf2d3b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/90bf2d3b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/90bf2d3b
Branch: refs/heads/branch-3.0
Commit: 90bf2d3b52245242cc0213ea58b42cdc706498a4
Parents: 2750808
Author: Rohith Sharma K S
Authored: Fri Aug 17 11:11:56 2018 +0530
Committer: Rohith Sharma K S
Committed: Fri Aug 17 11:14:10 2018 +0530
--
.../main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/90bf2d3b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 4ea6085..6c65b19 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1086,7 +1086,7 @@ public class YarnConfiguration extends Configuration {
NM_PREFIX + "collector-service.address";
public static final int DEFAULT_NM_COLLECTOR_SERVICE_PORT = 8048;
public static final String DEFAULT_NM_COLLECTOR_SERVICE_ADDRESS =
- "0.0.0.0:" + DEFAULT_NM_LOCALIZER_PORT;
+ "0.0.0.0:" + DEFAULT_NM_COLLECTOR_SERVICE_PORT;
/** Interval in between cache cleanups.*/
public static final String NM_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS =
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration. Contributed by Prabha Manepalli.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 5237bdfb5 -> 3532bd5c8
YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration.
Contributed by Prabha Manepalli.
(cherry picked from commit 1697a0230696e1ed6d9c19471463b44a6d791dfa)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3532bd5c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3532bd5c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3532bd5c
Branch: refs/heads/branch-3.1
Commit: 3532bd5c8b7a0593e67c0830b0d5ad10074309cf
Parents: 5237bdf
Author: Rohith Sharma K S
Authored: Fri Aug 17 11:11:56 2018 +0530
Committer: Rohith Sharma K S
Committed: Fri Aug 17 11:13:25 2018 +0530
--
.../main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3532bd5c/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 5f2f985..affa76a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1196,7 +1196,7 @@ public class YarnConfiguration extends Configuration {
NM_PREFIX + "collector-service.address";
public static final int DEFAULT_NM_COLLECTOR_SERVICE_PORT = 8048;
public static final String DEFAULT_NM_COLLECTOR_SERVICE_ADDRESS =
- "0.0.0.0:" + DEFAULT_NM_LOCALIZER_PORT;
+ "0.0.0.0:" + DEFAULT_NM_COLLECTOR_SERVICE_PORT;
/** Interval in between cache cleanups.*/
public static final String NM_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS =
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration. Contributed by Prabha Manepalli.
Repository: hadoop
Updated Branches:
refs/heads/branch-2 04c8b5dbb -> e2210a517
YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration.
Contributed by Prabha Manepalli.
(cherry picked from commit 1697a0230696e1ed6d9c19471463b44a6d791dfa)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e2210a51
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e2210a51
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e2210a51
Branch: refs/heads/branch-2
Commit: e2210a5175cca12605cbbb402f0a42cf1d815efe
Parents: 04c8b5d
Author: Rohith Sharma K S
Authored: Fri Aug 17 11:11:56 2018 +0530
Committer: Rohith Sharma K S
Committed: Fri Aug 17 11:17:30 2018 +0530
--
.../main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e2210a51/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 3f0c735..3b39e55 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1038,7 +1038,7 @@ public class YarnConfiguration extends Configuration {
NM_PREFIX + "collector-service.address";
public static final int DEFAULT_NM_COLLECTOR_SERVICE_PORT = 8048;
public static final String DEFAULT_NM_COLLECTOR_SERVICE_ADDRESS =
- "0.0.0.0:" + DEFAULT_NM_LOCALIZER_PORT;
+ "0.0.0.0:" + DEFAULT_NM_COLLECTOR_SERVICE_PORT;
/** Interval in between cache cleanups.*/
public static final String NM_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS =
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration. Contributed by Prabha Manepalli.
Repository: hadoop
Updated Branches:
refs/heads/trunk edeb2a356 -> 1697a0230
YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration.
Contributed by Prabha Manepalli.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1697a023
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1697a023
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1697a023
Branch: refs/heads/trunk
Commit: 1697a0230696e1ed6d9c19471463b44a6d791dfa
Parents: edeb2a3
Author: Rohith Sharma K S
Authored: Fri Aug 17 11:11:56 2018 +0530
Committer: Rohith Sharma K S
Committed: Fri Aug 17 11:12:10 2018 +0530
--
.../main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1697a023/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 84bfb55..78e28f7 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1216,7 +1216,7 @@ public class YarnConfiguration extends Configuration {
NM_PREFIX + "collector-service.address";
public static final int DEFAULT_NM_COLLECTOR_SERVICE_PORT = 8048;
public static final String DEFAULT_NM_COLLECTOR_SERVICE_ADDRESS =
- "0.0.0.0:" + DEFAULT_NM_LOCALIZER_PORT;
+ "0.0.0.0:" + DEFAULT_NM_COLLECTOR_SERVICE_PORT;
/** Interval in between cache cleanups.*/
public static final String NM_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS =
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[1/2] hadoop git commit: YARN-8603. [UI2] Latest run application should be listed first in the RM UI. Contributed by Akhil PB.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 619019ccc -> 4816d10dd
YARN-8603. [UI2] Latest run application should be listed first in the RM UI.
Contributed by Akhil PB.
(cherry picked from commit 33482d35e802ac3468317b844122b347fe486861)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/83ccb75b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/83ccb75b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/83ccb75b
Branch: refs/heads/branch-3.1
Commit: 83ccb75b918fd9643f31202c6d189099ffa99229
Parents: 619019c
Author: Rohith Sharma K S
Authored: Fri Aug 3 13:58:41 2018 +0530
Committer: Rohith Sharma K S
Committed: Fri Aug 3 15:33:24 2018 +0530
--
.../src/main/webapp/app/controllers/app-table-columns.js | 7 +++
.../src/main/webapp/app/controllers/yarn-apps/apps.js | 4 +++-
.../src/main/webapp/app/serializers/yarn-app.js | 2 +-
3 files changed, 11 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/83ccb75b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/app-table-columns.js
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/app-table-columns.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/app-table-columns.js
index 33e3bb7..552a157 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/app-table-columns.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/app-table-columns.js
@@ -20,6 +20,7 @@
import Ember from 'ember';
import ColumnDef from 'em-table/utils/column-definition';
import TableDef from 'em-table/utils/table-definition';
+import Converter from 'yarn-ui/utils/converter';
export default Ember.Controller.extend({
tableDefinition: TableDef.create({
@@ -83,6 +84,9 @@ export default Ember.Controller.extend({
headerTitle: 'Start Time',
contentPath: 'startTime',
facetType: null,
+ getCellContent: function(row) {
+return Converter.timeStampToDate(row.get('startTime'));
+ }
}, {
id: 'elTime',
headerTitle: 'Elapsed Time',
@@ -169,6 +173,9 @@ export default Ember.Controller.extend({
headerTitle: 'Started Time',
contentPath: 'startTime',
facetType: null,
+ getCellContent: function(row) {
+return Converter.timeStampToDate(row.get('startTime'));
+ }
}, {
id: 'finishTime',
headerTitle: 'Finished Time',
http://git-wip-us.apache.org/repos/asf/hadoop/blob/83ccb75b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-apps/apps.js
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-apps/apps.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-apps/apps.js
index 55d5a88..10965cf 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-apps/apps.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-apps/apps.js
@@ -25,7 +25,9 @@ export default AppTableController.extend({
tableDefinition: TableDefinition.create({
searchType: 'manual',
enableFaceting: true,
-rowCount: 25
+rowCount: 25,
+sortColumnId: 'stTime',
+sortOrder: 'desc'
}),
searchText: Ember.computed.alias('tableDefinition.searchText'),
sortColumnId: Ember.computed.alias('tableDefinition.sortColumnId'),
http://git-wip-us.apache.org/repos/asf/hadoop/blob/83ccb75b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app.js
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app.js
index 36d1260..f4de725 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app.js
@@ -42,7 +42,7 @@ export default DS.JSONAPISerializer.extend({
user: payload.user,
queue: payload.queue,
state: payload.state,
- startTime: Converter.timeStampToDate(payload.startedTime),
+ startTime: payload.startedTime, // will be formatted in em-table
elapsedTime: payload.elapsedTime,
finishedTime:
