[hadoop] branch trunk updated: HDFS-14799. Do Not Call Map containsKey In Conjunction with get. Contributed by hemanthboyina.
This is an automated email from the ASF dual-hosted git repository.
ayushsaxena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new e04b8a4 HDFS-14799. Do Not Call Map containsKey In Conjunction with
get. Contributed by hemanthboyina.
e04b8a4 is described below
commit e04b8a46c3088d13bf010f2959062e1440332bcc
Author: Ayush Saxena
AuthorDate: Sat Sep 14 10:39:20 2019 +0530
HDFS-14799. Do Not Call Map containsKey In Conjunction with get.
Contributed by hemanthboyina.
---
.../hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java | 10 ++
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
old mode 100644
new mode 100755
index 7c83792..247ea62
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
@@ -112,17 +112,11 @@ class InvalidateBlocks {
}
private LightWeightHashSet getBlocksSet(final DatanodeInfo dn) {
-if (nodeToBlocks.containsKey(dn)) {
- return nodeToBlocks.get(dn);
-}
-return null;
+return nodeToBlocks.get(dn);
}
private LightWeightHashSet getECBlocksSet(final DatanodeInfo dn) {
-if (nodeToECBlocks.containsKey(dn)) {
- return nodeToECBlocks.get(dn);
-}
-return null;
+return nodeToECBlocks.get(dn);
}
private LightWeightHashSet getBlocksSet(final DatanodeInfo dn,
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-2.9 updated: HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu.
This is an automated email from the ASF dual-hosted git repository.
ayushsaxena pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.9 by this push:
new 5b4e9a5 HDFS-14303. Addendum: check block directory logic not correct
when there is only meta file, print no meaning warn log. Contributed by qiang
Liu.
5b4e9a5 is described below
commit 5b4e9a5dc4b2e108a8eadbdca8e42ba772bfe46e
Author: Ayush Saxena
AuthorDate: Sat Sep 14 10:30:46 2019 +0530
HDFS-14303. Addendum: check block directory logic not correct when there is
only meta file, print no meaning warn log. Contributed by qiang Liu.
---
.../org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
index d4ff643..f4bf60a 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
@@ -1092,8 +1092,9 @@ public class TestDirectoryScanner {
@Test
public void testDirectoryScannerInFederatedCluster() throws Exception {
+HdfsConfiguration conf = new HdfsConfiguration(CONF);
//Create Federated cluster with two nameservices and one DN
-try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(CONF)
+try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
.nnTopology(MiniDFSNNTopology.simpleHAFederatedTopology(2))
.numDataNodes(1).build()) {
cluster.waitActive();
@@ -1110,7 +,7 @@ public class TestDirectoryScanner {
int bp2Files = 2;
writeFile(fs2, bp2Files);
//Call the Directory scanner
- scanner = new DirectoryScanner(dataNode, fds, CONF);
+ scanner = new DirectoryScanner(dataNode, fds, conf);
scanner.setRetainDiffs(true);
scanner.reconcile();
//Check blocks in corresponding BP
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-2 updated: HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu.
This is an automated email from the ASF dual-hosted git repository.
ayushsaxena pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new b9e832b HDFS-14303. Addendum: check block directory logic not correct
when there is only meta file, print no meaning warn log. Contributed by qiang
Liu.
b9e832b is described below
commit b9e832bf732cde0e4a17d857abc48bbeb04f0251
Author: Ayush Saxena
AuthorDate: Sat Sep 14 10:30:46 2019 +0530
HDFS-14303. Addendum: check block directory logic not correct when there is
only meta file, print no meaning warn log. Contributed by qiang Liu.
---
.../org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
index d4ff643..f4bf60a 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
@@ -1092,8 +1092,9 @@ public class TestDirectoryScanner {
@Test
public void testDirectoryScannerInFederatedCluster() throws Exception {
+HdfsConfiguration conf = new HdfsConfiguration(CONF);
//Create Federated cluster with two nameservices and one DN
-try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(CONF)
+try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
.nnTopology(MiniDFSNNTopology.simpleHAFederatedTopology(2))
.numDataNodes(1).build()) {
cluster.waitActive();
@@ -1110,7 +,7 @@ public class TestDirectoryScanner {
int bp2Files = 2;
writeFile(fs2, bp2Files);
//Call the Directory scanner
- scanner = new DirectoryScanner(dataNode, fds, CONF);
+ scanner = new DirectoryScanner(dataNode, fds, conf);
scanner.setRetainDiffs(true);
scanner.reconcile();
//Check blocks in corresponding BP
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.1 updated: HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu.
This is an automated email from the ASF dual-hosted git repository.
ayushsaxena pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new cab81f7 HDFS-14303. Addendum: check block directory logic not correct
when there is only meta file, print no meaning warn log. Contributed by qiang
Liu.
cab81f7 is described below
commit cab81f7f5fe579b927b27a1fe71bef1dd4857744
Author: Ayush Saxena
AuthorDate: Sat Sep 14 10:12:17 2019 +0530
HDFS-14303. Addendum: check block directory logic not correct when there is
only meta file, print no meaning warn log. Contributed by qiang Liu.
---
.../org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
index 2cbf49e..778707d 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
@@ -1131,8 +1131,9 @@ public class TestDirectoryScanner {
@Test
public void testDirectoryScannerInFederatedCluster() throws Exception {
+HdfsConfiguration conf = new HdfsConfiguration(CONF);
//Create Federated cluster with two nameservices and one DN
-try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(CONF)
+try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
.nnTopology(MiniDFSNNTopology.simpleHAFederatedTopology(2))
.numDataNodes(1).build()) {
cluster.waitActive();
@@ -1149,7 +1150,7 @@ public class TestDirectoryScanner {
int bp2Files = 2;
writeFile(fs2, bp2Files);
//Call the Directory scanner
- scanner = new DirectoryScanner(dataNode, fds, CONF);
+ scanner = new DirectoryScanner(dataNode, fds, conf);
scanner.setRetainDiffs(true);
scanner.reconcile();
//Check blocks in corresponding BP
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 02/02: HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien.
This is an automated email from the ASF dual-hosted git repository. weichiu pushed a commit to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hadoop.git commit 699297e0e6178bb802d86fe0f4b269fa35e717c7 Author: Yi Sheng AuthorDate: Sat Sep 14 02:11:04 2019 +0800 HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien. (cherry picked from commit 1843c4688a3040c48f6834d166494cbf68b5e4fd) Conflicts: LICENSE-binary (cherry picked from commit d39ebbfebc9da982cadfbe92f3fb6677e744622d) (cherry picked from commit 7d7b415409901bbb0095af4e9d56aade158c1438) --- hadoop-project/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 2626450..8948476 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -544,7 +544,7 @@ org.apache.commons commons-compress -1.18 +1.19 org.apache.commons - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 01/02: HADOOP-15804. upgrade to commons-compress 1.18. Contributed by Akira Ajisaka.
This is an automated email from the ASF dual-hosted git repository. weichiu pushed a commit to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hadoop.git commit 97491bfd7ff390a530a1e35a405d6e09c7284dfe Author: Takanobu Asanuma AuthorDate: Fri Oct 19 21:29:35 2018 +0900 HADOOP-15804. upgrade to commons-compress 1.18. Contributed by Akira Ajisaka. (cherry picked from commit 9bd18324c7801472409d9ad69ea365aa7a33a9c4) (cherry picked from commit bad5d81ac1472bf021463d7b6bdb166965ff021a) --- hadoop-project/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index ae2ab01..2626450 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -544,7 +544,7 @@ org.apache.commons commons-compress -1.4.1 +1.18 org.apache.commons - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-2 updated (c9a4630 -> 699297e)
This is an automated email from the ASF dual-hosted git repository. weichiu pushed a change to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hadoop.git. from c9a4630 YARN-9825. Changes for initializing placement rules with ResourceScheduler in branch-2. Contributed by Jonathan Hung. new 97491bf HADOOP-15804. upgrade to commons-compress 1.18. Contributed by Akira Ajisaka. new 699297e HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: hadoop-project/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu.
This is an automated email from the ASF dual-hosted git repository.
ayushsaxena pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 4126e67 HDFS-14303. Addendum: check block directory logic not correct
when there is only meta file, print no meaning warn log. Contributed by qiang
Liu.
4126e67 is described below
commit 4126e67e31d9431477aa39903bb03c0350f9c389
Author: Ayush Saxena
AuthorDate: Sat Sep 14 10:12:17 2019 +0530
HDFS-14303. Addendum: check block directory logic not correct when there is
only meta file, print no meaning warn log. Contributed by qiang Liu.
---
.../org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
index 438119a..13ae85e 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
@@ -1132,8 +1132,9 @@ public class TestDirectoryScanner {
@Test
public void testDirectoryScannerInFederatedCluster() throws Exception {
+HdfsConfiguration conf = new HdfsConfiguration(CONF);
//Create Federated cluster with two nameservices and one DN
-try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(CONF)
+try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
.nnTopology(MiniDFSNNTopology.simpleHAFederatedTopology(2))
.numDataNodes(1).build()) {
cluster.waitActive();
@@ -1150,7 +1151,7 @@ public class TestDirectoryScanner {
int bp2Files = 2;
writeFile(fs2, bp2Files);
//Call the Directory scanner
- scanner = new DirectoryScanner(dataNode, fds, CONF);
+ scanner = new DirectoryScanner(dataNode, fds, conf);
scanner.setRetainDiffs(true);
scanner.reconcile();
//Check blocks in corresponding BP
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated (6a9f7ca -> 9a931b8)
This is an automated email from the ASF dual-hosted git repository. elek pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 6a9f7ca Revert "HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message." add 9a931b8 HDDS-2125. maven-javadoc-plugin.version is missing in pom.ozone.xml No new revisions were added by this update. Summary of changes: pom.ozone.xml | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: Revert "HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message."
This is an automated email from the ASF dual-hosted git repository.
arp pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 6a9f7ca Revert "HDDS-2057. Incorrect Default OM Port in Ozone FS URI
Error Message."
6a9f7ca is described below
commit 6a9f7caef47c0ccacf778134d33e0c7547017323
Author: Arpit Agarwal
AuthorDate: Fri Sep 13 11:40:42 2019 -0700
Revert "HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message."
This reverts commit 95010a41fcea6ecf5dfd46d6e6f6f38c8b3e2a66.
---
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 26 +++---
1 file changed, 8 insertions(+), 18 deletions(-)
diff --git
a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
index a1648b4..1759e5c 100644
---
a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
+++
b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
@@ -43,7 +43,6 @@ import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathIsNotEmptyDirectoryException;
import org.apache.hadoop.fs.permission.FsPermission;
-import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.ozone.OmUtils;
import org.apache.hadoop.ozone.om.exceptions.OMException;
import org.apache.hadoop.security.UserGroupInformation;
@@ -88,20 +87,11 @@ public class BasicOzoneFileSystem extends FileSystem {
private static final Pattern URL_SCHEMA_PATTERN =
Pattern.compile("([^\\.]+)\\.([^\\.]+)\\.{0,1}(.*)");
- private OzoneConfiguration getOzoneConf(Configuration conf) {
-
-return (conf instanceof OzoneConfiguration) ?
-(OzoneConfiguration) conf : new OzoneConfiguration(conf);
- }
-
- private String getUriExceptionText(Configuration conf) {
-
-return "Ozone file system URL should be one of the following formats: "
-+ "o3fs://bucket.volume/key OR "
-+ "o3fs://bucket.volume.om-host.example.com/key OR "
-+ "o3fs://bucket.volume.om-host.example.com:"
-+ OmUtils.getOmRpcPort(getOzoneConf(conf)) + "/key";
- }
+ private static final String URI_EXCEPTION_TEXT = "Ozone file system URL " +
+ "should be one of the following formats: " +
+ "o3fs://bucket.volume/key OR " +
+ "o3fs://bucket.volume.om-host.example.com/key OR " +
+ "o3fs://bucket.volume.om-host.example.com:5678/key";
@Override
public void initialize(URI name, Configuration conf) throws IOException {
@@ -121,7 +111,7 @@ public class BasicOzoneFileSystem extends FileSystem {
Matcher matcher = URL_SCHEMA_PATTERN.matcher(authority);
if (!matcher.matches()) {
- throw new IllegalArgumentException(getUriExceptionText(conf));
+ throw new IllegalArgumentException(URI_EXCEPTION_TEXT);
}
String bucketStr = matcher.group(1);
String volumeStr = matcher.group(2);
@@ -133,14 +123,14 @@ public class BasicOzoneFileSystem extends FileSystem {
String[] parts = remaining.split(":");
// Array length should be either 1(hostname or service id) or
2(host:port)
if (parts.length > 2) {
-throw new IllegalArgumentException(getUriExceptionText(conf));
+throw new IllegalArgumentException(URI_EXCEPTION_TEXT);
}
omHost = parts[0];
if (parts.length == 2) {
try {
omPort = Integer.parseInt(parts[1]);
} catch (NumberFormatException e) {
- throw new IllegalArgumentException(getUriExceptionText(conf));
+ throw new IllegalArgumentException(URI_EXCEPTION_TEXT);
}
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDDS-2007. Make ozone fs shell command work with OM HA service ids (#1360)
This is an automated email from the ASF dual-hosted git repository.
bharat pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new e22a324 HDDS-2007. Make ozone fs shell command work with OM HA
service ids (#1360)
e22a324 is described below
commit e22a324f87a3b24868c4dd3975c94dcfb840fabc
Author: Siyao Meng <50227127+smen...@users.noreply.github.com>
AuthorDate: Fri Sep 13 11:22:00 2019 -0700
HDDS-2007. Make ozone fs shell command work with OM HA service ids (#1360)
---
.../hadoop/ozone/client/OzoneClientFactory.java| 69 +++-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 7 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 24 ++
.../ozone/om/ha/OMFailoverProxyProvider.java | 26 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 6 +-
.../src/main/compose/ozone-om-ha/docker-config | 11 +-
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 7 +
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 5 +
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 19 +-
.../org/apache/hadoop/ozone/RatisTestHelper.java | 2 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 2 +-
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 5 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 9 +-
.../snapshot/TestOzoneManagerSnapshotProvider.java | 7 +-
.../apache/hadoop/ozone/ozShell/TestS3Shell.java | 2 +-
.../hadoop/ozone/recon/ReconControllerModule.java | 2 +-
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 31 +-
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 10 +-
.../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 348 +
19 files changed, 553 insertions(+), 39 deletions(-)
diff --git
a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneClientFactory.java
b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneClientFactory.java
index 713a6b2..caf989e 100644
---
a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneClientFactory.java
+++
b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneClientFactory.java
@@ -119,6 +119,35 @@ public final class OzoneClientFactory {
* @param omRpcPort
*RPC port of OzoneManager.
*
+ * @param omServiceId
+ *Service ID of OzoneManager HA cluster.
+ *
+ * @param config
+ *Configuration to be used for OzoneClient creation
+ *
+ * @return OzoneClient
+ *
+ * @throws IOException
+ */
+ public static OzoneClient getRpcClient(String omHost, Integer omRpcPort,
+ String omServiceId, Configuration config) throws IOException {
+Preconditions.checkNotNull(omHost);
+Preconditions.checkNotNull(omRpcPort);
+Preconditions.checkNotNull(omServiceId);
+Preconditions.checkNotNull(config);
+config.set(OZONE_OM_ADDRESS_KEY, omHost + ":" + omRpcPort);
+return getRpcClient(omServiceId, config);
+ }
+
+ /**
+ * Returns an OzoneClient which will use RPC protocol.
+ *
+ * @param omHost
+ *hostname of OzoneManager to connect.
+ *
+ * @param omRpcPort
+ *RPC port of OzoneManager.
+ *
* @param config
*Configuration to be used for OzoneClient creation
*
@@ -139,6 +168,28 @@ public final class OzoneClientFactory {
/**
* Returns an OzoneClient which will use RPC protocol.
*
+ * @param omServiceId
+ *Service ID of OzoneManager HA cluster.
+ *
+ * @param config
+ *Configuration to be used for OzoneClient creation
+ *
+ * @return OzoneClient
+ *
+ * @throws IOException
+ */
+ public static OzoneClient getRpcClient(String omServiceId,
+ Configuration config) throws IOException {
+Preconditions.checkNotNull(omServiceId);
+Preconditions.checkNotNull(config);
+// Won't set OZONE_OM_ADDRESS_KEY here since service id is passed directly,
+// leaving OZONE_OM_ADDRESS_KEY value as is.
+return getClient(getClientProtocol(config, omServiceId), config);
+ }
+
+ /**
+ * Returns an OzoneClient which will use RPC protocol.
+ *
* @param config
*used for OzoneClient creation
*
@@ -185,8 +236,24 @@ public final class OzoneClientFactory {
*/
private static ClientProtocol getClientProtocol(Configuration config)
throws IOException {
+return getClientProtocol(config, null);
+ }
+
+ /**
+ * Returns an instance of Protocol class.
+ *
+ *
+ * @param config
+ *Configuration used to initialize ClientProtocol.
+ *
+ * @return ClientProtocol
+ *
+ * @throws IOException
+ */
+ private static ClientProtocol getClientProtocol(Configuration config,
+ String omServiceId) throws IOException {
try {
- return new RpcClient(config);
+ return new RpcClient(config, omServiceId);
} catch (Exception e) {
final String message
[hadoop] branch branch-3.2 updated: HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien.
This is an automated email from the ASF dual-hosted git repository. weichiu pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.2 by this push: new d39ebbf HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien. d39ebbf is described below commit d39ebbfebc9da982cadfbe92f3fb6677e744622d Author: Yi Sheng AuthorDate: Sat Sep 14 02:11:04 2019 +0800 HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien. (cherry picked from commit 1843c4688a3040c48f6834d166494cbf68b5e4fd) Conflicts: LICENSE-binary --- hadoop-project/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index e16b428..5b7f2d3 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -624,7 +624,7 @@ org.apache.commons commons-compress -1.18 +1.19 org.apache.commons - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.1 updated: HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien.
This is an automated email from the ASF dual-hosted git repository. weichiu pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.1 by this push: new 7d7b415 HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien. 7d7b415 is described below commit 7d7b415409901bbb0095af4e9d56aade158c1438 Author: Yi Sheng AuthorDate: Sat Sep 14 02:11:04 2019 +0800 HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien. (cherry picked from commit 1843c4688a3040c48f6834d166494cbf68b5e4fd) Conflicts: LICENSE-binary (cherry picked from commit d39ebbfebc9da982cadfbe92f3fb6677e744622d) --- hadoop-project/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index b433589..ff9e73f 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -598,7 +598,7 @@ org.apache.commons commons-compress -1.18 +1.19 org.apache.commons - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated (8024818 -> 1843c46)
This is an automated email from the ASF dual-hosted git repository. weichiu pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 8024818 HDDS-1786 : Datanodes takeSnapshot should delete previously created snapshots (#1163) add 1843c46 HADOOP-16555. Update commons-compress to 1.19. (#1425) Contributed by YiSheng Lien. No new revisions were added by this update. Summary of changes: LICENSE-binary | 2 +- hadoop-project/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated (06ad654 -> 8024818)
This is an automated email from the ASF dual-hosted git repository. shashikant pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 06ad654 HDFS-14821. Make HDFS-14617 (fsimage sub-sections) off by default. Contributed by Stephen O'Donnell. add 8024818 HDDS-1786 : Datanodes takeSnapshot should delete previously created snapshots (#1163) No new revisions were added by this update. Summary of changes: .../server/ratis/RatisServerConfiguration.java | 48 + .../transport/server/ratis/XceiverServerRatis.java | 15 -- .../client/rpc/TestContainerStateMachine.java | 62 +- 3 files changed, 120 insertions(+), 5 deletions(-) create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/RatisServerConfiguration.java - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 01/02: YARN-9787. Typo in analysesErrorMsg. Contributed by kevin su.
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 42390073499deb6dc66e1e89b55888dba1214cf9
Author: HUAN-PING SU
AuthorDate: Sun Sep 8 05:45:01 2019 +0800
YARN-9787. Typo in analysesErrorMsg. Contributed by kevin su.
Signed-off-by: Wei-Chiu Chuang
Reviewed-by: Surendra Singh Lilhore
---
.../server/nodemanager/containermanager/launcher/ContainerLaunch.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
index 8516543..41e26d4 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
@@ -785,7 +785,8 @@ public class ContainerLaunch implements Callable {
StringBuilder analysis = new StringBuilder();
if (errorMsg.indexOf("Error: Could not find or load main class"
+ " org.apache.hadoop.mapreduce") != -1) {
- analysis.append("Please check whether your etc/hadoop/mapred-site.xml "
+ analysis.append(
+ "Please check whether your /etc/hadoop/mapred-site.xml "
+ "contains the below configuration:\n");
analysis.append("\n")
.append(" yarn.app.mapreduce.am.env\n")
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated (95010a4 -> 06ad654)
This is an automated email from the ASF dual-hosted git repository. weichiu pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 95010a4 HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message. new 4239007 YARN-9787. Typo in analysesErrorMsg. Contributed by kevin su. new 06ad654 HDFS-14821. Make HDFS-14617 (fsimage sub-sections) off by default. Contributed by Stephen O'Donnell. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java| 2 +- .../hadoop-hdfs/src/main/resources/hdfs-default.xml| 7 ++- .../nodemanager/containermanager/launcher/ContainerLaunch.java | 3 ++- 3 files changed, 9 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 02/02: HDFS-14821. Make HDFS-14617 (fsimage sub-sections) off by default. Contributed by Stephen O'Donnell.
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 06ad6540b3cdb8fb957f900aaebc9d1d289fabaf
Author: Stephen O'Donnell
AuthorDate: Fri Sep 13 10:35:10 2019 -0700
HDFS-14821. Make HDFS-14617 (fsimage sub-sections) off by default.
Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang
---
.../src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java| 2 +-
.../hadoop-hdfs/src/main/resources/hdfs-default.xml| 7 ++-
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 6f76dc0..4b2058b 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -899,7 +899,7 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
public static final String DFS_IMAGE_PARALLEL_LOAD_KEY =
"dfs.image.parallel.load";
- public static final boolean DFS_IMAGE_PARALLEL_LOAD_DEFAULT = true;
+ public static final boolean DFS_IMAGE_PARALLEL_LOAD_DEFAULT = false;
public static final String DFS_IMAGE_PARALLEL_TARGET_SECTIONS_KEY =
"dfs.image.parallel.target.sections";
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
index 69017df..d2ca0af 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
@@ -1393,7 +1393,7 @@
dfs.image.parallel.load
- true
+ false
If true, write sub-section entries to the fsimage index so it can
be loaded in parallel. Also controls whether parallel loading
@@ -1403,6 +1403,11 @@
Parallel loading is not compatible with image compression,
so if dfs.image.compress is set to true this setting will be
ignored and no parallel loading will occur.
+Enabling this feature may impact rolling upgrades and downgrades if
+the previous version does not support this feature. If the feature was
+enabled and a downgrade is required, first set this parameter to
+false and then save the namespace to create a fsimage with no
+sub-sections and then perform the downgrade.
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message.
This is an automated email from the ASF dual-hosted git repository.
aengineer pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 95010a4 HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error
Message.
95010a4 is described below
commit 95010a41fcea6ecf5dfd46d6e6f6f38c8b3e2a66
Author: sdeka
AuthorDate: Thu Aug 29 21:23:50 2019 +0530
HDDS-2057. Incorrect Default OM Port in Ozone FS URI Error Message.
Signed-off-by: Anu Engineer
---
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 26 +++---
1 file changed, 18 insertions(+), 8 deletions(-)
diff --git
a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
index 4d7bfd9..06eedba 100644
---
a/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
+++
b/hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
@@ -43,6 +43,7 @@ import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathIsNotEmptyDirectoryException;
import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
import org.apache.hadoop.ozone.OmUtils;
import org.apache.hadoop.ozone.om.exceptions.OMException;
import org.apache.hadoop.security.UserGroupInformation;
@@ -87,11 +88,20 @@ public class BasicOzoneFileSystem extends FileSystem {
private static final Pattern URL_SCHEMA_PATTERN =
Pattern.compile("([^\\.]+)\\.([^\\.]+)\\.{0,1}(.*)");
- private static final String URI_EXCEPTION_TEXT = "Ozone file system URL " +
- "should be one of the following formats: " +
- "o3fs://bucket.volume/key OR " +
- "o3fs://bucket.volume.om-host.example.com/key OR " +
- "o3fs://bucket.volume.om-host.example.com:5678/key";
+ private OzoneConfiguration getOzoneConf(Configuration conf) {
+
+return (conf instanceof OzoneConfiguration) ?
+(OzoneConfiguration) conf : new OzoneConfiguration(conf);
+ }
+
+ private String getUriExceptionText(Configuration conf) {
+
+return "Ozone file system URL should be one of the following formats: "
++ "o3fs://bucket.volume/key OR "
++ "o3fs://bucket.volume.om-host.example.com/key OR "
++ "o3fs://bucket.volume.om-host.example.com:"
++ OmUtils.getOmRpcPort(getOzoneConf(conf)) + "/key";
+ }
@Override
public void initialize(URI name, Configuration conf) throws IOException {
@@ -106,7 +116,7 @@ public class BasicOzoneFileSystem extends FileSystem {
Matcher matcher = URL_SCHEMA_PATTERN.matcher(authority);
if (!matcher.matches()) {
- throw new IllegalArgumentException(URI_EXCEPTION_TEXT);
+ throw new IllegalArgumentException(getUriExceptionText(conf));
}
String bucketStr = matcher.group(1);
String volumeStr = matcher.group(2);
@@ -118,14 +128,14 @@ public class BasicOzoneFileSystem extends FileSystem {
String[] parts = remaining.split(":");
// Array length should be either 1(host) or 2(host:port)
if (parts.length > 2) {
-throw new IllegalArgumentException(URI_EXCEPTION_TEXT);
+throw new IllegalArgumentException(getUriExceptionText(conf));
}
omHost = parts[0];
if (parts.length == 2) {
try {
omPort = Integer.parseInt(parts[1]);
} catch (NumberFormatException e) {
- throw new IllegalArgumentException(URI_EXCEPTION_TEXT);
+ throw new IllegalArgumentException(getUriExceptionText(conf));
}
} else {
// If port number is not specified, read it from config
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch ozone-0.4.1 updated: HDDS-2122. Broken logo image on category sub-pages (#1437)
This is an automated email from the ASF dual-hosted git repository. nanda pushed a commit to branch ozone-0.4.1 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/ozone-0.4.1 by this push: new fd10837 HDDS-2122. Broken logo image on category sub-pages (#1437) fd10837 is described below commit fd10837a04dc7e1221fff1addc4c1695718b86e8 Author: Doroszlai, Attila AuthorDate: Fri Sep 13 16:11:00 2019 +0530 HDDS-2122. Broken logo image on category sub-pages (#1437) Signed-off-by: Nanda kumar (cherry picked from commit 4a9a6a21b8ebe6c762b1050a802cb7dd80f004da) --- hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html index 598ede6..0f26571 100644 --- a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html +++ b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html @@ -24,7 +24,7 @@ - + Apache Hadoop Ozone/HDDS documentation - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-2 updated: YARN-9825. Changes for initializing placement rules with ResourceScheduler in branch-2. Contributed by Jonathan Hung.
This is an automated email from the ASF dual-hosted git repository.
varunsaxena pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new c9a4630 YARN-9825. Changes for initializing placement rules with
ResourceScheduler in branch-2. Contributed by Jonathan Hung.
c9a4630 is described below
commit c9a46308a1bfac9278b19e179bf3132603d52e99
Author: Varun Saxena
AuthorDate: Fri Sep 13 17:03:47 2019 +0530
YARN-9825. Changes for initializing placement rules with ResourceScheduler
in branch-2. Contributed by Jonathan Hung.
---
.../resourcemanager/placement/PlacementRule.java | 9 ++--
.../placement/UserGroupMappingPlacementRule.java | 62 +-
.../scheduler/capacity/CapacityScheduler.java | 44 +--
3 files changed, 78 insertions(+), 37 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
index 47dc48a..805fcf7 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
@@ -18,20 +18,19 @@
package org.apache.hadoop.yarn.server.resourcemanager.placement;
-import java.util.Map;
+import java.io.IOException;
import org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext;
import org.apache.hadoop.yarn.exceptions.YarnException;
-import org.apache.hadoop.yarn.server.resourcemanager.RMContext;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
public abstract class PlacementRule {
public String getName() {
return this.getClass().getName();
}
- public void initialize(Map parameters, RMContext rmContext)
- throws YarnException {
- }
+ public abstract boolean initialize(
+ ResourceScheduler scheduler) throws IOException;
/**
* Get queue for a given application
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
index d617d16..bac419d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
@@ -32,6 +32,14 @@ import org.apache.hadoop.yarn.exceptions.YarnException;
import
org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.QueueMapping.MappingType;
import com.google.common.annotations.VisibleForTesting;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueue;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerContext;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue;
+
public class UserGroupMappingPlacementRule extends PlacementRule {
private static final Log LOG = LogFactory
@@ -95,6 +103,10 @@ public class UserGroupMappingPlacementRule extends
PlacementRule {
}
}
+ public UserGroupMappingPlacementRule(){
+this(false, null, null);
+ }
+
public UserGroupMappingPlacementRule(boolean overrideWithQueueMappings,
List newMappings, Groups groups) {
this.mappings = newMappings;
@@ -156,7 +168,55 @@ public class UserGroupMappingPlacementRule extends
PlacementRule {
return queueName;
}
-
+
+ @VisibleForTesting
+ @Override
+ public boolean initialize(ResourceScheduler scheduler)
+ throws IOException {
+if (!(scheduler instanceof CapacityScheduler)) {
+ throw new IOException(
+
[hadoop] branch trunk updated: HADOOP-16562. [pb-upgrade] Update docker image to have 3.7.1 protoc executable (#1429).
This is an automated email from the ASF dual-hosted git repository. vinayakumarb pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 39e82ac HADOOP-16562. [pb-upgrade] Update docker image to have 3.7.1 protoc executable (#1429). 39e82ac is described below commit 39e82acc485db0e66cbb3dd26b59dfe111ce6a10 Author: Vinayakumar B AuthorDate: Fri Sep 13 15:30:24 2019 +0530 HADOOP-16562. [pb-upgrade] Update docker image to have 3.7.1 protoc executable (#1429). Addendum patch. Moved protobuf-3.7.1 installation within YETUS marker. --- dev-support/docker/Dockerfile | 20 ++-- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/dev-support/docker/Dockerfile b/dev-support/docker/Dockerfile index fe38395..371bdde 100644 --- a/dev-support/docker/Dockerfile +++ b/dev-support/docker/Dockerfile @@ -200,6 +200,16 @@ RUN curl -L -s -S \ ### ENV MAVEN_OPTS -Xms256m -Xmx1536m +RUN mkdir -p /opt/protobuf-3.7-src \ +&& curl -L -s -S \ + https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ + -o /opt/protobuf-3.7.1.tar.gz \ +&& tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C /opt/protobuf-3.7-src \ +&& cd /opt/protobuf-3.7-src \ +&& ./configure --prefix=/opt/protobuf-3.7 \ +&& make install \ +&& cd /root \ +&& rm -rf /opt/protobuf-3.7-src ### # Everything past this point is either not needed for testing or breaks Yetus. @@ -217,16 +227,6 @@ RUN curl -L -o hugo.deb https://github.com/gohugoio/hugo/releases/download/v0.30 # Keep 2.5.0 as well, until 3.7.1 upgrade is complete. ## # hadolint ignore=DL3003 -RUN mkdir -p /opt/protobuf-3.7-src \ -&& curl -L -s -S \ - https://github.com/protocolbuffers/protobuf/releases/download/v3.7.1/protobuf-java-3.7.1.tar.gz \ - -o /opt/protobuf-3.7.1.tar.gz \ -&& tar xzf /opt/protobuf-3.7.1.tar.gz --strip-components 1 -C /opt/protobuf-3.7-src \ -&& cd /opt/protobuf-3.7-src \ -&& ./configure --prefix=/opt/protobuf-3.7 \ -&& make install \ -&& cd /root \ -&& rm -rf /opt/protobuf-3.7-src # Add a welcome message and environment checks. COPY hadoop_env_checks.sh /root/hadoop_env_checks.sh - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDDS-2122. Broken logo image on category sub-pages (#1437)
This is an automated email from the ASF dual-hosted git repository. nanda pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 4a9a6a2 HDDS-2122. Broken logo image on category sub-pages (#1437) 4a9a6a2 is described below commit 4a9a6a21b8ebe6c762b1050a802cb7dd80f004da Author: Doroszlai, Attila AuthorDate: Fri Sep 13 16:11:00 2019 +0530 HDDS-2122. Broken logo image on category sub-pages (#1437) Signed-off-by: Nanda kumar --- hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html index 598ede6..0f26571 100644 --- a/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html +++ b/hadoop-hdds/docs/themes/ozonedoc/layouts/partials/navbar.html @@ -24,7 +24,7 @@ - + Apache Hadoop Ozone/HDDS documentation - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
