[hadoop-ozone] branch master updated (2420ee8 -> fdcc696)
This is an automated email from the ASF dual-hosted git repository. umamahesh pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git. from 2420ee8 HDDS-4288. the icon of hadoop-ozone is bigger than ever (#1452) add fdcc696 HDDS-4287: Exclude protobuff classes from ozone-filesystem-hadoop3 jars (#1455). Contributed by Uma Maheswara Rao G. No new revisions were added by this update. Summary of changes: hadoop-ozone/ozonefs-shaded/pom.xml | 4 1 file changed, 4 insertions(+) - To unsubscribe, e-mail: ozone-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-commits-h...@hadoop.apache.org
[hadoop-ozone] branch master updated: HDDS-4288. the icon of hadoop-ozone is bigger than ever (#1452)
This is an automated email from the ASF dual-hosted git repository.
vivekratnavel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
The following commit(s) were added to refs/heads/master by this push:
new 2420ee8 HDDS-4288. the icon of hadoop-ozone is bigger than ever
(#1452)
2420ee8 is described below
commit 2420ee83b5cb06e86aedf00284fc438f7a3976c4
Author: Elek, Márton
AuthorDate: Tue Sep 29 20:47:33 2020 +0200
HDDS-4288. the icon of hadoop-ozone is bigger than ever (#1452)
---
hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html | 4 ++--
hadoop-hdds/docs/themes/ozonedoc/static/css/ozonedoc.css | 11 +++
2 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html
b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html
index 155f7ad..f942e4a 100644
--- a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html
+++ b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html
@@ -23,8 +23,8 @@
-
-
+
+
Apache Hadoop Ozone/HDDS documentation
diff --git a/hadoop-hdds/docs/themes/ozonedoc/static/css/ozonedoc.css
b/hadoop-hdds/docs/themes/ozonedoc/static/css/ozonedoc.css
index 1e6061f..aa57c92 100644
--- a/hadoop-hdds/docs/themes/ozonedoc/static/css/ozonedoc.css
+++ b/hadoop-hdds/docs/themes/ozonedoc/static/css/ozonedoc.css
@@ -54,6 +54,16 @@ a:hover {
border: 0;
}
+A.ozone-logo {
+ height: 50px;
+ padding: 5px 10px 5px 0px;
+}
+
+.ozone-logo img {
+ width: 40px;
+ padding: 0px;
+}
+
/*
* Sidebar
*/
@@ -62,6 +72,7 @@ a:hover {
.sidebar {
display: none;
}
+
@media (min-width: 768px) {
.sidebar {
position: fixed;
-
To unsubscribe, e-mail: ozone-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-commits-h...@hadoop.apache.org
[hadoop-ozone] branch master updated: HDDS-4288. the icon of hadoop-ozone is bigger than ever (#1452)
This is an automated email from the ASF dual-hosted git repository.
vivekratnavel pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
The following commit(s) were added to refs/heads/master by this push:
new 2420ee8 HDDS-4288. the icon of hadoop-ozone is bigger than ever
(#1452)
2420ee8 is described below
commit 2420ee83b5cb06e86aedf00284fc438f7a3976c4
Author: Elek, Márton
AuthorDate: Tue Sep 29 20:47:33 2020 +0200
HDDS-4288. the icon of hadoop-ozone is bigger than ever (#1452)
---
hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html | 4 ++--
hadoop-hdds/docs/themes/ozonedoc/static/css/ozonedoc.css | 11 +++
2 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html
b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html
index 155f7ad..f942e4a 100644
--- a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html
+++ b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html
@@ -23,8 +23,8 @@
-
-
+
+
Apache Hadoop Ozone/HDDS documentation
diff --git a/hadoop-hdds/docs/themes/ozonedoc/static/css/ozonedoc.css
b/hadoop-hdds/docs/themes/ozonedoc/static/css/ozonedoc.css
index 1e6061f..aa57c92 100644
--- a/hadoop-hdds/docs/themes/ozonedoc/static/css/ozonedoc.css
+++ b/hadoop-hdds/docs/themes/ozonedoc/static/css/ozonedoc.css
@@ -54,6 +54,16 @@ a:hover {
border: 0;
}
+A.ozone-logo {
+ height: 50px;
+ padding: 5px 10px 5px 0px;
+}
+
+.ozone-logo img {
+ width: 40px;
+ padding: 0px;
+}
+
/*
* Sidebar
*/
@@ -62,6 +72,7 @@ a:hover {
.sidebar {
display: none;
}
+
@media (min-width: 768px) {
.sidebar {
position: fixed;
-
To unsubscribe, e-mail: ozone-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-commits-h...@hadoop.apache.org
[hadoop-ozone] branch HDDS-3698-upgrade updated: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions. (#1430)
This is an automated email from the ASF dual-hosted git repository.
avijayan pushed a commit to branch HDDS-3698-upgrade
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
The following commit(s) were added to refs/heads/HDDS-3698-upgrade by this push:
new a2788b7 HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that
applies all committed Ratis transactions. (#1430)
a2788b7 is described below
commit a2788b72cf068ab1723cd1c64670acca4db78e8f
Author: avijayanhwx <14299376+avijayan...@users.noreply.github.com>
AuthorDate: Tue Sep 29 09:41:43 2020 -0700
HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all
committed Ratis transactions. (#1430)
---
hadoop-hdds/common/pom.xml | 8 ++
.../hadoop/hdds/ratis/RatisUpgradeUtils.java | 96 +
.../hadoop/hdds/ratis/TestRatisUpgradeUtils.java | 97 ++
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 4 +
.../apache/hadoop/ozone/om/OMStarterInterface.java | 2 +
.../org/apache/hadoop/ozone/om/OzoneManager.java | 64 --
.../hadoop/ozone/om/OzoneManagerStarter.java | 38 +
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 2 +-
.../ozone/om/ratis/OzoneManagerRatisServer.java| 4 +
.../hadoop/ozone/om/TestOzoneManagerStarter.java | 11 +++
10 files changed, 319 insertions(+), 7 deletions(-)
diff --git a/hadoop-hdds/common/pom.xml b/hadoop-hdds/common/pom.xml
index cc171f1..d7abf0e 100644
--- a/hadoop-hdds/common/pom.xml
+++ b/hadoop-hdds/common/pom.xml
@@ -188,6 +188,14 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd;>
hamcrest-all
test
+
+ com.codahale.metrics
+ metrics-core
+ 3.0.2
+ test
+
+
+
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisUpgradeUtils.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisUpgradeUtils.java
new file mode 100644
index 000..796668d
--- /dev/null
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisUpgradeUtils.java
@@ -0,0 +1,96 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.hdds.ratis;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.ratis.protocol.RaftGroup;
+import org.apache.ratis.server.impl.RaftServerImpl;
+import org.apache.ratis.server.impl.RaftServerProxy;
+import org.apache.ratis.statemachine.StateMachine;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Ratis utility functions.
+ */
+public final class RatisUpgradeUtils {
+
+ private RatisUpgradeUtils() {
+ }
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(RatisUpgradeUtils.class);
+
+ /**
+ * Flush all committed transactions in a given Raft Server for a given group.
+ * @param stateMachine state machine to use
+ * @param raftGroup raft group
+ * @param server Raft server proxy instance.
+ * @param maxTimeToWaitSeconds Max time to wait before declaring failure.
+ * @throws InterruptedException when interrupted
+ * @throws IOException on error while waiting
+ */
+ public static void waitForAllTxnsApplied(
+ StateMachine stateMachine,
+ RaftGroup raftGroup,
+ RaftServerProxy server,
+ long maxTimeToWaitSeconds,
+ long timeBetweenRetryInSeconds)
+ throws InterruptedException, IOException {
+
+long intervalTime = TimeUnit.SECONDS.toMillis(timeBetweenRetryInSeconds);
+long endTime = System.currentTimeMillis() +
+TimeUnit.SECONDS.toMillis(maxTimeToWaitSeconds);
+boolean success = false;
+while (System.currentTimeMillis() < endTime) {
+ success = checkIfAllTransactionsApplied(stateMachine, server, raftGroup);
+ if (success) {
+break;
+ }
+ Thread.sleep(intervalTime);
+}
+
+if (!success) {
+ throw new IOException(String.format("After waiting for %d seconds, " +
+ "State Machine has not applied all the transactions.",
+ maxTimeToWaitSeconds));
+}
+
+long snapshotIndex = stateMachine.takeSnapshot();
+if (snapshotIndex !=
[hadoop-ozone] branch master updated (68642c2 -> 3ad1034)
This is an automated email from the ASF dual-hosted git repository. elek pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git. from 68642c2 HDDS-4023. Delete closed container after all blocks have been deleted. (#1338) add 3ad1034 HDDS-4215. Update Freon doc in source tree. (#1403) No new revisions were added by this update. Summary of changes: hadoop-hdds/docs/content/tools/TestTools.md| 12 ++-- hadoop-hdds/docs/content/tools/TestTools.zh.md | 12 ++-- 2 files changed, 20 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: ozone-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-commits-h...@hadoop.apache.org
[hadoop-ozone] branch master updated (34f3b91 -> 68642c2)
This is an automated email from the ASF dual-hosted git repository. sammichen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git. from 34f3b91 HDDS-4232. Use single thread for KeyDeletingService. (#1415) add 68642c2 HDDS-4023. Delete closed container after all blocks have been deleted. (#1338) No new revisions were added by this update. Summary of changes: .../container/AbstractContainerReportHandler.java | 51 +--- .../hdds/scm/container/ContainerReportHandler.java | 12 +-- .../IncrementalContainerReportHandler.java | 2 +- .../hdds/scm/container/ReplicationManager.java | 96 +- .../hdds/scm/container/SCMContainerManager.java| 6 +- .../hdds/scm/safemode/ContainerSafeModeRule.java | 4 +- .../java/org/apache/hadoop/hdds/scm/TestUtils.java | 1 + .../scm/container/TestContainerReportHandler.java | 33 .../commandhandler/TestBlockDeletion.java | 41 - .../ReconIncrementalContainerReportHandler.java| 2 +- 10 files changed, 219 insertions(+), 29 deletions(-) - To unsubscribe, e-mail: ozone-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-commits-h...@hadoop.apache.org
[hadoop-ozone] branch master updated (6267a39 -> 34f3b91)
This is an automated email from the ASF dual-hosted git repository. adoroszlai pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git. from 6267a39 HDDS-4231. Background Service blocks on task results. (#1414) add 34f3b91 HDDS-4232. Use single thread for KeyDeletingService. (#1415) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/hadoop/ozone/om/KeyDeletingService.java| 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: ozone-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-commits-h...@hadoop.apache.org
[hadoop-ozone] branch master updated (525ecbb -> 6267a39)
This is an automated email from the ASF dual-hosted git repository. adoroszlai pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git. from 525ecbb HDDS-3727. Volume space: check quotaUsageInBytes when write key. (#1434) add 6267a39 HDDS-4231. Background Service blocks on task results. (#1414) No new revisions were added by this update. Summary of changes: .../hadoop/hdds/utils/BackgroundService.java | 73 +++--- .../apache/hadoop/hdds/utils/BackgroundTask.java | 4 +- .../hadoop/hdds/utils/BackgroundTaskQueue.java | 5 +- .../background/BlockDeletingService.java | 3 +- .../container/common/TestBlockDeletingService.java | 3 +- .../hdds/scm/block/SCMBlockDeletingService.java| 3 +- .../apache/hadoop/ozone/om/KeyDeletingService.java | 3 +- .../hadoop/ozone/om/OpenKeyCleanupService.java | 3 +- 8 files changed, 35 insertions(+), 62 deletions(-) - To unsubscribe, e-mail: ozone-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-commits-h...@hadoop.apache.org
[hadoop-ozone] branch HDDS-2823 updated: HDDS-4115. CLI command to show current SCM leader and follower status. (#1346)
This is an automated email from the ASF dual-hosted git repository.
licheng pushed a commit to branch HDDS-2823
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git
The following commit(s) were added to refs/heads/HDDS-2823 by this push:
new 1c2a950 HDDS-4115. CLI command to show current SCM leader and
follower status. (#1346)
1c2a950 is described below
commit 1c2a9502783c677dc24679f879f11782e7a201c1
Author: Rui Wang
AuthorDate: Mon Sep 28 23:57:11 2020 -0700
HDDS-4115. CLI command to show current SCM leader and follower status.
(#1346)
* HDDS-4115. CLI command to show current SCM leader and follower status.
* fixup! address comments.
* fixup! fix style
* fixup! fix style
* fixup! add cli command description
* fixup! 1. use `ozone admin scm roles` 2. fix acceptance test
* fixup! fix checkstyle
* fixup! address comments
---
.../java/org/apache/hadoop/hdds/scm/ScmInfo.java | 32 -
.../apache/hadoop/hdds/scm/client/ScmClient.java | 5 +-
...inerLocationProtocolClientSideTranslatorPB.java | 4 +-
.../interface-client/src/main/proto/hdds.proto | 1 +
.../apache/hadoop/hdds/scm/ha/SCMHAManager.java| 6 ++
.../hadoop/hdds/scm/ha/SCMHAManagerImpl.java | 11 +++
...inerLocationProtocolServerSideTranslatorPB.java | 2 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 3 +-
.../hdds/scm/server/StorageContainerManager.java | 4 ++
.../hadoop/hdds/scm/ha/MockSCMHAManager.java | 9 +++
.../hdds/scm/cli/ContainerOperationClient.java | 4 ++
.../dist/src/main/smoketest/admincli/scmha.robot | 28
.../apache/hadoop/ozone/shell/TestScmAdminHA.java | 79 ++
.../org/apache/hadoop/ozone/admin/OzoneAdmin.java | 2 +
.../admin/scm/GetScmRatisRolesSubcommand.java | 46 +
.../apache/hadoop/ozone/admin/scm/ScmAdmin.java| 60
.../hadoop/ozone/admin/scm/package-info.java | 22 ++
17 files changed, 312 insertions(+), 6 deletions(-)
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmInfo.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmInfo.java
index 6236feb..b9d823e 100644
--- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmInfo.java
+++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmInfo.java
@@ -18,6 +18,9 @@
package org.apache.hadoop.hdds.scm;
+import java.util.ArrayList;
+import java.util.List;
+
/**
* ScmInfo wraps the result returned from SCM#getScmInfo which
* contains clusterId and the SCM Id.
@@ -25,6 +28,7 @@ package org.apache.hadoop.hdds.scm;
public final class ScmInfo {
private String clusterId;
private String scmId;
+ private List peerRoles;
/**
* Builder for ScmInfo.
@@ -32,6 +36,11 @@ public final class ScmInfo {
public static class Builder {
private String clusterId;
private String scmId;
+private List peerRoles;
+
+public Builder() {
+ peerRoles = new ArrayList<>();
+}
/**
* sets the cluster id.
@@ -53,14 +62,25 @@ public final class ScmInfo {
return this;
}
+/**
+ * Set peer address in Scm HA.
+ * @param roles ratis peer address in the format of [ip|hostname]:port
+ * @return Builder for scmInfo
+ */
+public Builder setRatisPeerRoles(List roles) {
+ peerRoles.addAll(roles);
+ return this;
+}
+
public ScmInfo build() {
- return new ScmInfo(clusterId, scmId);
+ return new ScmInfo(clusterId, scmId, peerRoles);
}
}
- private ScmInfo(String clusterId, String scmId) {
+ private ScmInfo(String clusterId, String scmId, List peerRoles) {
this.clusterId = clusterId;
this.scmId = scmId;
+this.peerRoles = peerRoles;
}
/**
@@ -78,4 +98,12 @@ public final class ScmInfo {
public String getScmId() {
return scmId;
}
+
+ /**
+ * Gets the list of peer roles (currently address) in Scm HA.
+ * @return List of peer address
+ */
+ public List getRatisPeerRoles() {
+return peerRoles;
+ }
}
diff --git
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java
index e4369fa..7c3c94c 100644
---
a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java
+++
b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java
@@ -246,5 +246,8 @@ public interface ScmClient extends Closeable {
*/
boolean getReplicationManagerStatus() throws IOException;
-
+ /**
+ * returns the list of ratis peer roles. Currently only include peer address.
+ */
+ List getScmRatisRoles() throws IOException;
}
diff --git
a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java
