[hadoop] branch trunk updated: HDFS-14558. RBF: Isolation/Fairness documentation. Contributed by Fengnan Li.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 0d7ac54 HDFS-14558. RBF: Isolation/Fairness documentation. Contributed by Fengnan Li. 0d7ac54 is described below commit 0d7ac54510fbed8957d053546d606d5084e3e708 Author: Yiqun Lin AuthorDate: Tue Jan 12 23:38:55 2021 +0800 HDFS-14558. RBF: Isolation/Fairness documentation. Contributed by Fengnan Li. --- .../src/site/markdown/HDFSRouterFederation.md | 23 ++ 1 file changed, 23 insertions(+) diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md index 66f039a..702fa44 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md @@ -184,6 +184,18 @@ Router relies on a state store to distribute tokens across all routers. Apart fr See the Apache JIRA ticket [HDFS-13532](https://issues.apache.org/jira/browse/HDFS-13532) for more information on this feature. +### Isolation +Router supports assignment of the dedicated number of RPC handlers to achieve isolation for all downstream nameservices it is configured to proxy. Since large or busy clusters may have relatively higher RPC traffic to the namenode compared to other clusters namenodes, this feature if enabled allows admins to configure higher number of RPC handlers for busy clusters. If dedicated handlers are not assigned for specific nameservices, equal distribution of RPC handlers is done for all config [...] + +If a downstream namenode is slow/busy enough that permits are unavailable, routers would throw StandByException exception to the client. This would in turn trigger a failover behavior at the client side and clients would connect to a different router in the cluster. This offers a positive effect of automatically load balancing RPCs across all routers nodes. This is important to ensure that a single router does not become a bottleneck in case of unhealthy namenodes and all handlers availa [...] + +Users can configure handlers based on steady state load that individual downstream namenodes expect and can introduce more routers to the cluster to handle more RPCs overall. Because of the bouncing behavior that clients automatically get in this feature in an event where certain namenodes are overloaded, good clients connecting to good namenodes will always continue to use Rpc lanes dedicated to them. For bad behaving namenodes or backfill jobs that put spiky loads on namenodes, more ro [...] + +Overall the isolation feature is exposed via a configuration dfs.federation.router.handler.isolation.enable. The default value of this feature will be “false”. Users can also introduce their own fairness policy controller for custom allocation of handlers to various nameservices. + +See the Apache JIRA ticket [HDFS-14090](https://issues.apache.org/jira/browse/HDFS-14090) for more information on this feature. + + Deployment -- @@ -482,6 +494,17 @@ Kerberos and Delegation token supported in federation. | dfs.federation.router.kerberos.internal.spnego.principal | `${dfs.web.authentication.kerberos.principal}` | The server principal used by the Router for web UI SPNEGO authentication when Kerberos security is enabled. This is typically set to HTTP/_h...@realm.tld The SPNEGO server principal begins with the prefix HTTP/ by convention. If the value is '*', the web server will attempt to login with every principal specified in the keytab file 'dfs.web.authentication.kerberos.keytab'. | | dfs.federation.router.secret.manager.class | `org.apache.hadoop.hdfs.server.federation.router.security.token.ZKDelegationTokenSecretManagerImpl` | Class to implement state store to delegation tokens. Default implementation uses zookeeper as the backend to store delegation tokens. | +### Isolation + +Isolation and dedicated assignment of RPC handlers across all configured downstream nameservices. The sum of these numbers must be strictly smaller than the total number of router handlers (configed by dfs.federation.router.handler.count). + +| Property | Default | Description| +|: |: |: | +| dfs.federation.router.fairness.enable | `false` | If `true`, dedicated RPC handlers will be assigned to each nameservice based on the fairness assignment policy configured. | +| dfs.federation.router.fairness.policy.controller.class | `org.apache.hadoop.hdfs.server.federation.fairness.NoRouterRpcFairnessPolicyController` | Default handler allocation model to be used if isolation feature is enabled. Recommend to use `org.apache.hadoop.hdfs.server.federation.fairness.StaticRouterRpcFairnessPolicyContr
[hadoop] branch branch-2.9 updated: HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch branch-2.9 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-2.9 by this push: new f70fc91 HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. f70fc91 is described below commit f70fc91a0218829ecd432cbe83d28ee0bd686644 Author: Yiqun Lin AuthorDate: Mon Dec 7 18:52:12 2020 +0800 HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. (cherry picked from commit da1ea2530fa61c53a99770e10889023c474fb4ef) --- hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto index c07dd9e..1b82272 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto @@ -180,7 +180,7 @@ message StorageTypeQuotaInfosProto { } message StorageTypeQuotaInfoProto { - required StorageTypeProto type = 1; + optional StorageTypeProto type = 1 [default = DISK]; required uint64 quota = 2; required uint64 consumed = 3; } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-2.10 updated: HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch branch-2.10 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-2.10 by this push: new 8ddd85b HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. 8ddd85b is described below commit 8ddd85b0e6fc41bac854c910aecb5cbc0392535a Author: Yiqun Lin AuthorDate: Mon Dec 7 18:52:12 2020 +0800 HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. (cherry picked from commit da1ea2530fa61c53a99770e10889023c474fb4ef) --- hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto index c07dd9e..1b82272 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto @@ -180,7 +180,7 @@ message StorageTypeQuotaInfosProto { } message StorageTypeQuotaInfoProto { - required StorageTypeProto type = 1; + optional StorageTypeProto type = 1 [default = DISK]; required uint64 quota = 2; required uint64 consumed = 3; } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.0 updated: HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new b46ec61 HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. b46ec61 is described below commit b46ec6190968205a83015fdc0450b5ce55b03fbb Author: Yiqun Lin AuthorDate: Mon Dec 7 18:52:12 2020 +0800 HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. (cherry picked from commit da1ea2530fa61c53a99770e10889023c474fb4ef) --- hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto index 7b3f5ed..df770c8 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto @@ -182,7 +182,7 @@ message StorageTypeQuotaInfosProto { } message StorageTypeQuotaInfoProto { - required StorageTypeProto type = 1; + optional StorageTypeProto type = 1 [default = DISK]; required uint64 quota = 2; required uint64 consumed = 3; } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new da1ea25 HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. da1ea25 is described below commit da1ea2530fa61c53a99770e10889023c474fb4ef Author: Yiqun Lin AuthorDate: Mon Dec 7 18:52:12 2020 +0800 HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. --- hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto index 5ae58cf..08fce71 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto @@ -198,7 +198,7 @@ message StorageTypeQuotaInfosProto { } message StorageTypeQuotaInfoProto { - required StorageTypeProto type = 1; + optional StorageTypeProto type = 1 [default = DISK]; required uint64 quota = 2; required uint64 consumed = 3; } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDFS-15640. Add diff threshold to FedBalance. Contributed by Jinglun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 15a5f53 HDFS-15640. Add diff threshold to FedBalance. Contributed by Jinglun. 15a5f53 is described below commit 15a5f5367366fdd76933d0ff6499363fcbc8873e Author: Yiqun Lin AuthorDate: Tue Oct 27 10:41:10 2020 +0800 HDFS-15640. Add diff threshold to FedBalance. Contributed by Jinglun. --- .../hadoop/tools/fedbalance/DistCpProcedure.java | 43 +- .../apache/hadoop/tools/fedbalance/FedBalance.java | 22 ++- .../hadoop/tools/fedbalance/FedBalanceContext.java | 21 +++ .../hadoop/tools/fedbalance/FedBalanceOptions.java | 11 ++ .../src/site/markdown/HDFSFederationBalance.md | 1 + .../tools/fedbalance/TestDistCpProcedure.java | 34 - 6 files changed, 119 insertions(+), 13 deletions(-) diff --git a/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java b/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java index 223b777..33d37be 100644 --- a/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java +++ b/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/DistCpProcedure.java @@ -89,6 +89,8 @@ public class DistCpProcedure extends BalanceProcedure { private boolean forceCloseOpenFiles; /* Disable write by setting the mount point readonly. */ private boolean useMountReadOnly; + /* The threshold of diff entries. */ + private int diffThreshold; private FsPermission fPerm; // the permission of the src. private AclStatus acl; // the acl of the src. @@ -134,6 +136,7 @@ public class DistCpProcedure extends BalanceProcedure { this.bandWidth = context.getBandwidthLimit(); this.forceCloseOpenFiles = context.getForceCloseOpenFiles(); this.useMountReadOnly = context.getUseMountReadOnly(); +this.diffThreshold = context.getDiffThreshold(); srcFs = (DistributedFileSystem) context.getSrc().getFileSystem(conf); dstFs = (DistributedFileSystem) context.getDst().getFileSystem(conf); } @@ -227,12 +230,8 @@ public class DistCpProcedure extends BalanceProcedure { } else { throw new RetryException(); // wait job complete. } -} else if (!verifyDiff()) { - if (!verifyOpenFiles() || forceCloseOpenFiles) { -updateStage(Stage.DISABLE_WRITE); - } else { -throw new RetryException(); - } +} else if (diffDistCpStageDone()) { + updateStage(Stage.DISABLE_WRITE); } else { submitDiffDistCp(); } @@ -372,14 +371,38 @@ public class DistCpProcedure extends BalanceProcedure { } /** - * Verify whether the src has changed since CURRENT_SNAPSHOT_NAME snapshot. + * Check whether the conditions are satisfied for moving to the next stage. + * If the diff entries size is no greater than the threshold and the open + * files could be force closed or there is no open file, then moving to the + * next stage. + * + * @return true if moving to the next stage. false if the conditions are not + * satisfied. + * @throws RetryException if the conditions are not satisfied and the diff + * size is under the given threshold scope. + */ + @VisibleForTesting + boolean diffDistCpStageDone() throws IOException, RetryException { +int diffSize = getDiffSize(); +if (diffSize <= diffThreshold) { + if (forceCloseOpenFiles || !verifyOpenFiles()) { +return true; + } else { +throw new RetryException(); + } +} +return false; + } + + /** + * Get number of the diff entries. * - * @return true if the src has changed. + * @return number of the diff entries. */ - private boolean verifyDiff() throws IOException { + private int getDiffSize() throws IOException { SnapshotDiffReport diffReport = srcFs.getSnapshotDiffReport(src, CURRENT_SNAPSHOT_NAME, ""); -return diffReport.getDiffList().size() > 0; +return diffReport.getDiffList().size(); } /** diff --git a/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/FedBalance.java b/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/FedBalance.java index ca6b0df..c850798 100644 --- a/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/FedBalance.java +++ b/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/FedBalance.java @@ -51,6 +51,7 @@ import static org.apache.hadoop.tools.fedbalance.FedBalanceOptions.BANDWIDTH; import static org.apache.hadoop.tools.fedbalance.FedBalanceOptions.TRASH
[hadoop] branch trunk updated: HDFS-15374. Add documentation for fedbalance tool. Contributed by Jinglun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new ff8bb67 HDFS-15374. Add documentation for fedbalance tool. Contributed by Jinglun. ff8bb67 is described below commit ff8bb672000980f3de7391e5d268e789d5cbe974 Author: Yiqun Lin AuthorDate: Wed Jul 1 14:18:18 2020 +0800 HDFS-15374. Add documentation for fedbalance tool. Contributed by Jinglun. --- hadoop-project/src/site/site.xml | 1 + .../src/site/markdown/HDFSFederationBalance.md | 171 + .../src/site/resources/css/site.css| 30 .../resources/images/BalanceProcedureScheduler.png | Bin 0 -> 48275 bytes 4 files changed, 202 insertions(+) diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index c3c0f19..4c9d356 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -198,6 +198,7 @@ + diff --git a/hadoop-tools/hadoop-federation-balance/src/site/markdown/HDFSFederationBalance.md b/hadoop-tools/hadoop-federation-balance/src/site/markdown/HDFSFederationBalance.md new file mode 100644 index 000..ff42eaf --- /dev/null +++ b/hadoop-tools/hadoop-federation-balance/src/site/markdown/HDFSFederationBalance.md @@ -0,0 +1,171 @@ + + +HDFS Federation Balance Guide += + + + +Overview + + + HDFS Federation Balance is a tool balancing data across different federation + namespaces. It uses [DistCp](../hadoop-distcp/DistCp.html) to copy data from + the source path to the target path. First it creates a snapshot at the source + path and submits the initial distcp. Second it uses distcp diff to do the + incremental copy until the source and the target are the same. Then If it's + working in RBF mode it updates the mount table in Router. Finally it moves the + source to trash. + + This document aims to describe the usage and design of the HDFS Federation + Balance. + +Usage +- + +### Basic Usage + + The hdfs federation balance tool supports both normal federation cluster and + router-based federation cluster. Taking rbf for example. Supposing we have a + mount entry in Router: + +Source Destination +/foo/src hdfs://namespace-0/foo/src + + The command below runs an hdfs federation balance job. The first parameter is + the mount entry. The second one is the target path which must include the + target cluster. The option `-router` indicates this is in router-based + federation mode. + +bash$ /bin/hadoop fedbalance -router submit /foo/src hdfs://namespace-1/foo/dst + + It copies data from hdfs://namespace-0/foo/src to hdfs://namespace-1/foo/dst + incrementally and finally updates the mount entry to: + +Source Destination +/foo/src hdfs://namespace-1/foo/dst + + If the hadoop shell process exits unexpectedly, we can use the command below + to continue the unfinished job: + +bash$ /bin/hadoop fedbalance continue + + This will scan the journal to find all the unfinished jobs, recover and + continue to execute them. + + If we want to balance in a normal federation cluster, use the command below. + +bash$ /bin/hadoop fedbalance submit hdfs://namespace-0/foo/src hdfs://namespace-1/foo/dst + + In normal federation mode the source path must includes the path schema. + +### RBF Mode And Normal Federation Mode + + The hdfs federation balance tool has 2 modes: + + * the router-based federation mode (RBF mode). + * the normal federation mode. + + By default the command runs in the normal federation mode. You can specify the + rbf mode by using the option `-router`. + + In the rbf mode the first parameter is taken as the mount point. It disables + write by setting the mount point readonly. + + In the normal federation mode the first parameter is taken as the full path of + the source. The first parameter must include the source cluster. It disables + write by cancelling all the permissions of the source path. + + Details about disabling write see [HDFS FedBalance](#HDFS_FedBalance). + +### Command Options + +Command `submit` has 5 options: + +| Option key | Description | Default | +| -- | | --- | +| -router | Run in router-based federation mode. | Normal federation mode. | +| -forceCloseOpen | Force close all open files when there is no diff in the DIFF_DISTCP stage. | Wait until there is no open files. | +| -map | Max number of concurrent maps to use for copy. | 10 | +| -bandwidth | Specify bandwidth per map in MB. | 10 | +| -delay | Specify the delayed duration(millie seconds) when the job needs to retry. | 1000 | +| -moveToTrash | This options has 3 values
[hadoop] branch trunk updated: HDFS-15410. Add separated config file hdfs-fedbalance-default.xml for fedbalance tool. Contributed by Jinglun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new de2cb86 HDFS-15410. Add separated config file hdfs-fedbalance-default.xml for fedbalance tool. Contributed by Jinglun. de2cb86 is described below commit de2cb8626016f22b388da7796082b2e160059cf6 Author: Yiqun Lin AuthorDate: Wed Jul 1 14:06:27 2020 +0800 HDFS-15410. Add separated config file hdfs-fedbalance-default.xml for fedbalance tool. Contributed by Jinglun. --- .../apache/hadoop/tools/fedbalance/FedBalance.java | 48 ++ .../hadoop/tools/fedbalance/FedBalanceConfigs.java | 17 ...pBalanceOptions.java => FedBalanceOptions.java} | 16 .../procedure/BalanceProcedureScheduler.java | 7 +--- .../src/main/resources/hdfs-fedbalance-default.xml | 41 ++ .../hadoop/tools/fedbalance/TestFedBalance.java| 34 +++ 6 files changed, 125 insertions(+), 38 deletions(-) diff --git a/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/FedBalance.java b/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/FedBalance.java index adfb40b..8252957 100644 --- a/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/FedBalance.java +++ b/hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/FedBalance.java @@ -17,6 +17,7 @@ */ package org.apache.hadoop.tools.fedbalance; +import com.google.common.annotations.VisibleForTesting; import org.apache.commons.cli.CommandLine; import org.apache.commons.cli.CommandLineParser; import org.apache.commons.cli.GnuParser; @@ -25,7 +26,6 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hdfs.HdfsConfiguration; import org.apache.hadoop.tools.fedbalance.procedure.BalanceProcedure; import org.apache.hadoop.hdfs.server.federation.resolver.MountTableManager; import org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys; @@ -34,7 +34,6 @@ import org.apache.hadoop.hdfs.server.federation.store.records.MountTable; import org.apache.hadoop.tools.fedbalance.procedure.BalanceJob; import org.apache.hadoop.tools.fedbalance.procedure.BalanceProcedureScheduler; import org.apache.hadoop.net.NetUtils; -import org.apache.hadoop.util.ReflectionUtils; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; import org.slf4j.Logger; @@ -45,14 +44,13 @@ import java.net.InetSocketAddress; import java.util.Collection; import java.util.concurrent.TimeUnit; -import static org.apache.hadoop.tools.fedbalance.DistCpBalanceOptions.ROUTER; -import static org.apache.hadoop.tools.fedbalance.DistCpBalanceOptions.FORCE_CLOSE_OPEN; -import static org.apache.hadoop.tools.fedbalance.DistCpBalanceOptions.MAP; -import static org.apache.hadoop.tools.fedbalance.DistCpBalanceOptions.BANDWIDTH; -import static org.apache.hadoop.tools.fedbalance.DistCpBalanceOptions.TRASH; -import static org.apache.hadoop.tools.fedbalance.DistCpBalanceOptions.DELAY_DURATION; -import static org.apache.hadoop.tools.fedbalance.DistCpBalanceOptions.CLI_OPTIONS; -import static org.apache.hadoop.tools.fedbalance.FedBalanceConfigs.FEDERATION_BALANCE_CLASS; +import static org.apache.hadoop.tools.fedbalance.FedBalanceOptions.ROUTER; +import static org.apache.hadoop.tools.fedbalance.FedBalanceOptions.FORCE_CLOSE_OPEN; +import static org.apache.hadoop.tools.fedbalance.FedBalanceOptions.MAP; +import static org.apache.hadoop.tools.fedbalance.FedBalanceOptions.BANDWIDTH; +import static org.apache.hadoop.tools.fedbalance.FedBalanceOptions.TRASH; +import static org.apache.hadoop.tools.fedbalance.FedBalanceOptions.DELAY_DURATION; +import static org.apache.hadoop.tools.fedbalance.FedBalanceOptions.CLI_OPTIONS; import static org.apache.hadoop.tools.fedbalance.FedBalanceConfigs.TrashOption; /** @@ -73,6 +71,10 @@ public class FedBalance extends Configured implements Tool { private static final String MOUNT_TABLE_PROCEDURE = "mount-table-procedure"; private static final String TRASH_PROCEDURE = "trash-procedure"; + private static final String FED_BALANCE_DEFAULT_XML = + "hdfs-fedbalance-default.xml"; + private static final String FED_BALANCE_SITE_XML = "hdfs-fedbalance-site.xml"; + /** * This class helps building the balance job. */ @@ -210,7 +212,7 @@ public class FedBalance extends Configured implements Tool { public int run(String[] args) throws Exception { CommandLineParser parser = new GnuParser(); CommandLine command = -parser.parse(DistCpBalanceOptions.CLI_OPTIONS, args, true); +parser.parse(FedBalanceOptions.CLI_OPTIONS, args, true);
[hadoop] branch trunk updated: HDFS-15346. FedBalance tool implementation. Contributed by Jinglun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 9cbd76c HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 9cbd76c is described below commit 9cbd76cc775b58dfedb943f971b3307ec5702f13 Author: Yiqun Lin AuthorDate: Thu Jun 18 13:33:25 2020 +0800 HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. --- .../src/main/resources/assemblies/hadoop-tools.xml | 15 + .../apache/hadoop/hdfs/protocol/HdfsConstants.java | 2 + hadoop-project/pom.xml | 17 + hadoop-tools/hadoop-federation-balance/pom.xml | 249 .../tools/fedbalance/DistCpBalanceOptions.java | 95 +++ .../hadoop/tools/fedbalance/DistCpProcedure.java | 635 + .../apache/hadoop/tools/fedbalance/FedBalance.java | 377 .../hadoop/tools/fedbalance/FedBalanceConfigs.java | 19 +- .../hadoop/tools/fedbalance/FedBalanceContext.java | 286 ++ .../tools/fedbalance/MountTableProcedure.java | 244 .../hadoop/tools/fedbalance/TrashProcedure.java| 112 .../hadoop/tools/fedbalance}/package-info.java | 14 +- .../tools/fedbalance}/procedure/BalanceJob.java| 2 +- .../fedbalance}/procedure/BalanceJournal.java | 2 +- .../procedure/BalanceJournalInfoHDFS.java | 8 +- .../fedbalance}/procedure/BalanceProcedure.java| 4 +- .../procedure/BalanceProcedureScheduler.java | 8 +- .../tools/fedbalance}/procedure/package-info.java | 2 +- .../shellprofile.d/hadoop-federation-balance.sh| 38 ++ .../tools/fedbalance/TestDistCpProcedure.java | 446 +++ .../tools/fedbalance/TestMountTableProcedure.java | 222 +++ .../tools/fedbalance/TestTrashProcedure.java | 102 .../fedbalance}/procedure/MultiPhaseProcedure.java | 2 +- .../fedbalance}/procedure/RecordProcedure.java | 2 +- .../fedbalance}/procedure/RetryProcedure.java | 2 +- .../procedure/TestBalanceProcedureScheduler.java | 7 +- .../procedure/UnrecoverableProcedure.java | 2 +- .../tools/fedbalance}/procedure/WaitProcedure.java | 2 +- hadoop-tools/hadoop-tools-dist/pom.xml | 5 + hadoop-tools/pom.xml | 1 + 30 files changed, 2887 insertions(+), 35 deletions(-) diff --git a/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml b/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml index 054d8c0..db744f5 100644 --- a/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml +++ b/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml @@ -48,6 +48,14 @@ 0755 + ../hadoop-federation-balance/src/main/shellprofile.d + +* + + /libexec/shellprofile.d + 0755 + + ../hadoop-extras/src/main/shellprofile.d * @@ -112,6 +120,13 @@ + ../hadoop-federation-balance/target + /share/hadoop/${hadoop.component}/sources + +*-sources.jar + + + ../hadoop-extras/target /share/hadoop/${hadoop.component}/sources diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java index ab61e50..a025b9b 100755 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java @@ -105,6 +105,8 @@ public final class HdfsConstants { public static final String DOT_SNAPSHOT_DIR = ".snapshot"; public static final String SEPARATOR_DOT_SNAPSHOT_DIR = Path.SEPARATOR + DOT_SNAPSHOT_DIR; + public static final String DOT_SNAPSHOT_DIR_SEPARATOR = + DOT_SNAPSHOT_DIR + Path.SEPARATOR; public static final String SEPARATOR_DOT_SNAPSHOT_DIR_SEPARATOR = Path.SEPARATOR + DOT_SNAPSHOT_DIR + Path.SEPARATOR; public final static String DOT_RESERVED_STRING = ".reserved"; diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index f3a3d76..48928b5 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -329,6 +329,12 @@ org.apache.hadoop +hadoop-hdfs-rbf +${hadoop.version} +test-jar + + +org.apache.hadoop hadoop-mapreduce-client-app ${hadoop.version} @@ -580,6 +586,17 @@ org.apache.hadoop +hadoop-federation-balance +${hadoop.version} + + +org.apache.hadoop +hadoop-federation-balance +${hadoop.versio
[hadoop] branch trunk updated: HDFS-15340. RBF: Implement BalanceProcedureScheduler basic framework. Contributed by Jinglun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 1983eea HDFS-15340. RBF: Implement BalanceProcedureScheduler basic framework. Contributed by Jinglun. 1983eea is described below commit 1983eea62def58fb769f44c1d41dc29690274809 Author: Yiqun Lin AuthorDate: Wed May 20 10:39:40 2020 +0800 HDFS-15340. RBF: Implement BalanceProcedureScheduler basic framework. Contributed by Jinglun. --- .../apache/hadoop/hdfs/procedure/BalanceJob.java | 361 + .../hadoop/hdfs/procedure/BalanceJournal.java | 48 +++ .../hdfs/procedure/BalanceJournalInfoHDFS.java | 203 ++ .../hadoop/hdfs/procedure/BalanceProcedure.java| 226 +++ .../hdfs/procedure/BalanceProcedureConfigKeys.java | 41 ++ .../hdfs/procedure/BalanceProcedureScheduler.java | 450 .../apache/hadoop/hdfs/procedure/package-info.java | 29 ++ .../hadoop/hdfs/procedure/MultiPhaseProcedure.java | 88 .../hadoop/hdfs/procedure/RecordProcedure.java | 45 ++ .../hadoop/hdfs/procedure/RetryProcedure.java | 66 +++ .../procedure/TestBalanceProcedureScheduler.java | 451 + .../hdfs/procedure/UnrecoverableProcedure.java | 56 +++ .../hadoop/hdfs/procedure/WaitProcedure.java | 77 13 files changed, 2141 insertions(+) diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/procedure/BalanceJob.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/procedure/BalanceJob.java new file mode 100644 index 000..847092a --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/procedure/BalanceJob.java @@ -0,0 +1,361 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdfs.procedure; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.commons.lang3.builder.EqualsBuilder; +import org.apache.commons.lang3.builder.HashCodeBuilder; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.io.Writable; +import org.apache.hadoop.util.ReflectionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; +import java.util.Map; +import java.util.HashMap; +import java.util.Set; +import java.util.HashSet; +import java.util.List; +import java.util.ArrayList; + +/** + * A Job is a state machine consists of many procedures. The procedures are + * executed as a chain. Each procedure needs to specify the next procedure. If + * there is no next procedure then the job is finished. + */ +public final class BalanceJob implements Writable { + private String id; + private BalanceProcedureScheduler scheduler; + private volatile boolean jobDone = false; + private Exception error; + public static final Logger LOG = LoggerFactory.getLogger(BalanceJob.class); + private Map procedureTable = new HashMap<>(); + private T firstProcedure; + private T curProcedure; + private T lastProcedure; + private boolean removeAfterDone; + + static final String NEXT_PROCEDURE_NONE = "NONE"; + private static Set reservedNames = new HashSet<>(); + + static { +reservedNames.add(NEXT_PROCEDURE_NONE); + } + + public static class Builder { + +private List procedures = new ArrayList<>(); +private boolean removeAfterDone = false; + +/** + * Append a procedure to the tail. + */ +public Builder nextProcedure(T procedure) { + int size = procedures.size(); + if (size > 0) { +procedures.get(size - 1).setNextProcedure(procedure.name()); + } + procedure.setNextProcedure(NEXT_PROCEDURE_NONE); + procedures.add(procedure); + return this; +} + +/** + * Automatically remove this job from the scheduler cache when the job is + * done. + */ +public Builder removeAfterDone(boolean remove) { + removeAfterDone = remove; + return this; +}
[hadoop] branch branch-2.10 updated: HDFS-15264. Backport Datanode detection to branch-2.10. Contributed by Lisheng Sun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch branch-2.10 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-2.10 by this push: new e6374f0 HDFS-15264. Backport Datanode detection to branch-2.10. Contributed by Lisheng Sun. e6374f0 is described below commit e6374f031af6fb3a5467ccd12d6a4c8d7b0dae1e Author: Yiqun Lin AuthorDate: Sun May 17 11:59:10 2020 +0800 HDFS-15264. Backport Datanode detection to branch-2.10. Contributed by Lisheng Sun. --- .../java/org/apache/hadoop/hdfs/ClientContext.java | 49 ++ .../java/org/apache/hadoop/hdfs/DFSClient.java | 115 .../org/apache/hadoop/hdfs/DFSInputStream.java | 100 ++-- .../org/apache/hadoop/hdfs/DeadNodeDetector.java | 586 + .../hadoop/hdfs/client/HdfsClientConfigKeys.java | 39 ++ .../hadoop/hdfs/client/impl/DfsClientConf.java | 14 + .../src/main/resources/hdfs-default.xml| 73 +++ .../apache/hadoop/hdfs/TestDeadNodeDetection.java | 406 ++ 8 files changed, 1349 insertions(+), 33 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java index a31945c..2cb30f5 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java @@ -39,6 +39,7 @@ import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.net.NetworkTopology; import org.apache.hadoop.net.NodeBase; import org.apache.hadoop.net.ScriptBasedMapping; +import org.apache.hadoop.util.Daemon; import org.apache.hadoop.util.ReflectionUtils; import com.google.common.annotations.VisibleForTesting; @@ -117,6 +118,19 @@ public class ClientContext { private NodeBase clientNode; private boolean topologyResolutionEnabled; + private Daemon deadNodeDetectorThr = null; + + /** + * The switch to DeadNodeDetector. + */ + private boolean deadNodeDetectionEnabled = false; + + /** + * Detect the dead datanodes in advance, and share this information among all + * the DFSInputStreams in the same client. + */ + private DeadNodeDetector deadNodeDetector = null; + private ClientContext(String name, DfsClientConf conf, Configuration config) { final ShortCircuitConf scConf = conf.getShortCircuitConf(); @@ -133,6 +147,12 @@ public class ClientContext { this.byteArrayManager = ByteArrayManager.newInstance( conf.getWriteByteArrayManagerConf()); +this.deadNodeDetectionEnabled = conf.isDeadNodeDetectionEnabled(); +if (deadNodeDetectionEnabled && deadNodeDetector == null) { + deadNodeDetector = new DeadNodeDetector(name, config); + deadNodeDetectorThr = new Daemon(deadNodeDetector); + deadNodeDetectorThr.start(); +} initTopologyResolution(config); } @@ -250,4 +270,33 @@ public class ClientContext { datanodeInfo.getNetworkLocation()); return NetworkTopology.getDistanceByPath(clientNode, node); } + + /** + * The switch to DeadNodeDetector. If true, DeadNodeDetector is available. + */ + public boolean isDeadNodeDetectionEnabled() { +return deadNodeDetectionEnabled; + } + + /** + * Obtain DeadNodeDetector of the current client. + */ + public DeadNodeDetector getDeadNodeDetector() { +return deadNodeDetector; + } + + /** + * Close dead node detector thread. + */ + public void stopDeadNodeDetectorThread() { +if (deadNodeDetectorThr != null) { + deadNodeDetectorThr.interrupt(); + try { +deadNodeDetectorThr.join(); + } catch (InterruptedException e) { +LOG.warn("Encountered exception while waiting to join on dead " + +"node detector thread.", e); + } +} + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java index 77ee893..ad4e499 100755 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java @@ -48,6 +48,8 @@ import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.Random; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.SynchronousQueue; import java.util.concurrent.ThreadLocalRandom; import java.util.concurrent.ThreadPoolExecutor; @@ -240,6 +242,20 @@ public class DFSClient implements java.io.Closeable, RemotePeerFactory, private final int smallBufferSize; private final long serverDefaultsValidityPeriod;
[hadoop] branch trunk updated: HDFS-13811. RBF: Race condition between router admin quota update and periodic quota update service. Contributed by Jinglun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 47fdae7 HDFS-13811. RBF: Race condition between router admin quota update and periodic quota update service. Contributed by Jinglun. 47fdae7 is described below commit 47fdae79041ba2bb036ef7723a93ade5b1ac3619 Author: Yiqun Lin AuthorDate: Wed Dec 4 18:19:11 2019 +0800 HDFS-13811. RBF: Race condition between router admin quota update and periodic quota update service. Contributed by Jinglun. --- .../hdfs/server/federation/router/Router.java | 8 ++ .../federation/router/RouterQuotaManager.java | 21 .../router/RouterQuotaUpdateService.java | 40 +-- .../server/federation/store/MountTableStore.java | 12 ++- .../federation/store/impl/MountTableStoreImpl.java | 15 +++ .../server/federation/router/TestRouterQuota.java | 115 +++-- 6 files changed, 166 insertions(+), 45 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Router.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Router.java index a03d8d4..64fdabe 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Router.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Router.java @@ -43,6 +43,7 @@ import org.apache.hadoop.hdfs.server.federation.metrics.RBFMetrics; import org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics; import org.apache.hadoop.hdfs.server.federation.resolver.ActiveNamenodeResolver; import org.apache.hadoop.hdfs.server.federation.resolver.FileSubclusterResolver; +import org.apache.hadoop.hdfs.server.federation.store.MountTableStore; import org.apache.hadoop.hdfs.server.federation.store.RouterStore; import org.apache.hadoop.hdfs.server.federation.store.StateStoreService; import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem; @@ -292,6 +293,13 @@ public class Router extends CompositeService implements } super.serviceInit(conf); + +// Set quota manager in mount store to update quota usage in mount table. +if (stateStore != null) { + MountTableStore mountstore = + this.stateStore.getRegisteredRecordStore(MountTableStore.class); + mountstore.setQuotaManager(this.quotaManager); +} } private String getDisabledDependentServices() { diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaManager.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaManager.java index c1a5146..ceb758e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaManager.java @@ -170,6 +170,27 @@ public class RouterQuotaManager { } /** + * Update quota in cache. The usage will be preserved. + * @param path Mount table path. + * @param quota Corresponding quota value. + */ + public void updateQuota(String path, RouterQuotaUsage quota) { +writeLock.lock(); +try { + RouterQuotaUsage.Builder builder = new RouterQuotaUsage.Builder() + .quota(quota.getQuota()).spaceQuota(quota.getSpaceQuota()); + RouterQuotaUsage current = this.cache.get(path); + if (current != null) { +builder.fileAndDirectoryCount(current.getFileAndDirectoryCount()) +.spaceConsumed(current.getSpaceConsumed()); + } + this.cache.put(path, builder.build()); +} finally { + writeLock.unlock(); +} + } + + /** * Remove the entity from cache. * @param path Mount table path. */ diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaUpdateService.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaUpdateService.java index e5d4472..f1a86bf 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaUpdateService.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterQuotaUpdateService.java @@ -35,15 +35,13 @@ import org.apache.hadoop.hdfs.server.federation.resolver.RemoteLocation; import org.apache.hadoop.hdfs.server.federation.store.MountTableStore; import org.apache.hadoop.hdfs.server.federation.store.protocol.GetMountTableEntriesRequest; import
[hadoop] branch trunk updated: HDFS-15019. Refactor the unit test of TestDeadNodeDetection. Contributed by Lisheng Sun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new c3659f8 HDFS-15019. Refactor the unit test of TestDeadNodeDetection. Contributed by Lisheng Sun. c3659f8 is described below commit c3659f8f94bef7cfad0c3fb04391a7ffd4221679 Author: Yiqun Lin AuthorDate: Thu Nov 28 14:41:49 2019 +0800 HDFS-15019. Refactor the unit test of TestDeadNodeDetection. Contributed by Lisheng Sun. --- .../apache/hadoop/hdfs/TestDeadNodeDetection.java | 38 -- 1 file changed, 7 insertions(+), 31 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDeadNodeDetection.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDeadNodeDetection.java index 58f6d5d..a1e53cd 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDeadNodeDetection.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDeadNodeDetection.java @@ -51,6 +51,11 @@ public class TestDeadNodeDetection { public void setUp() { cluster = null; conf = new HdfsConfiguration(); +conf.setBoolean(DFS_CLIENT_DEAD_NODE_DETECTION_ENABLED_KEY, true); +conf.setLong( +DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_DEAD_NODE_INTERVAL_MS_KEY, 1000); +conf.setLong( +DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_SUSPECT_NODE_INTERVAL_MS_KEY, 100); } @After @@ -62,15 +67,6 @@ public class TestDeadNodeDetection { @Test public void testDeadNodeDetectionInBackground() throws Exception { -conf = new HdfsConfiguration(); -conf.setBoolean(DFS_CLIENT_DEAD_NODE_DETECTION_ENABLED_KEY, true); - conf.setLong(DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_DEAD_NODE_INTERVAL_MS_KEY, -1000); -conf.setLong( -DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_SUSPECT_NODE_INTERVAL_MS_KEY, 100); -// We'll be using a 512 bytes block size just for tests -// so making sure the checksum bytes match it too. -conf.setInt("io.bytes.per.checksum", 512); cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build(); cluster.waitActive(); @@ -123,10 +119,6 @@ public class TestDeadNodeDetection { @Test public void testDeadNodeDetectionInMultipleDFSInputStream() throws IOException { -conf.setBoolean(DFS_CLIENT_DEAD_NODE_DETECTION_ENABLED_KEY, true); -// We'll be using a 512 bytes block size just for tests -// so making sure the checksum bytes match it too. -conf.setInt("io.bytes.per.checksum", 512); cluster = new MiniDFSCluster.Builder(conf).numDataNodes(1).build(); cluster.waitActive(); @@ -149,8 +141,9 @@ public class TestDeadNodeDetection { } catch (BlockMissingException e) { } - din2 = (DFSInputStream) in1.getWrappedStream(); + din2 = (DFSInputStream) in2.getWrappedStream(); dfsClient2 = din2.getDFSClient(); + assertEquals(dfsClient1.toString(), dfsClient2.toString()); assertEquals(1, dfsClient1.getDeadNodes(din1).size()); assertEquals(1, dfsClient2.getDeadNodes(din2).size()); assertEquals(1, dfsClient1.getClientContext().getDeadNodeDetector() @@ -180,12 +173,6 @@ public class TestDeadNodeDetection { @Test public void testDeadNodeDetectionDeadNodeRecovery() throws Exception { -conf.setBoolean(DFS_CLIENT_DEAD_NODE_DETECTION_ENABLED_KEY, true); - conf.setLong(DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_DEAD_NODE_INTERVAL_MS_KEY, -1000); -// We'll be using a 512 bytes block size just for tests -// so making sure the checksum bytes match it too. -conf.setInt("io.bytes.per.checksum", 512); cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build(); cluster.waitActive(); @@ -228,13 +215,7 @@ public class TestDeadNodeDetection { @Test public void testDeadNodeDetectionMaxDeadNodesProbeQueue() throws Exception { -conf.setBoolean(DFS_CLIENT_DEAD_NODE_DETECTION_ENABLED_KEY, true); - conf.setLong(DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_DEAD_NODE_INTERVAL_MS_KEY, -1000); conf.setInt(DFS_CLIENT_DEAD_NODE_DETECTION_DEAD_NODE_QUEUE_MAX_KEY, 1); -// We'll be using a 512 bytes block size just for tests -// so making sure the checksum bytes match it too. -conf.setInt("io.bytes.per.checksum", 512); cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build(); cluster.waitActive(); @@ -268,12 +249,7 @@ public class TestDeadNodeDetection { @Test public void testDeadNodeDetectionSuspectNode() throws Exception { -conf = new HdfsConfiguration(); -conf.setBoolean(DFS_CLIENT_DEAD_NODE_DETECTION_ENABLED_KEY, true); conf.setInt(DFS_CLIENT_DEAD_NODE_DETECTION_SUSPECT_NODE_QUEUE_MAX_KEY, 1); -// We'll be using a 512 by
[hadoop] branch branch-2.10 updated: HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch branch-2.10 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-2.10 by this push: new 1a83415 HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 1a83415 is described below commit 1a834157602069ed82e29e380e1d660a10592daa Author: Yiqun Lin AuthorDate: Thu Nov 28 10:43:35 2019 +0800 HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. (cherry picked from commit 2b452b4e6063072b2bec491edd3f412eb7ac21f3) --- .../org/apache/hadoop/fs/CachingGetSpaceUsed.java | 34 +++-- .../server/datanode/fsdataset/FsDatasetSpi.java| 6 +++ .../datanode/fsdataset/impl/FsDatasetImpl.java | 12 ++--- .../fsdataset/impl/ReplicaCachingGetSpaceUsed.java | 1 + .../impl/TestReplicaCachingGetSpaceUsed.java | 55 ++ 5 files changed, 98 insertions(+), 10 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java index 92476d7..58dc82d 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java @@ -47,6 +47,7 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { private final long jitter; private final String dirPath; private Thread refreshUsed; + private boolean shouldFirstRefresh; /** * This is the constructor used by the builder. @@ -79,16 +80,30 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { this.refreshInterval = interval; this.jitter = jitter; this.used.set(initialUsed); +this.shouldFirstRefresh = true; } void init() { if (used.get() < 0) { used.set(0); + if (!shouldFirstRefresh) { +// Skip initial refresh operation, so we need to do first refresh +// operation immediately in refresh thread. +initRefeshThread(true); +return; + } refresh(); } +initRefeshThread(false); + } + /** + * RunImmediately should set true, if we skip the first refresh. + * @param runImmediately The param default should be false. + */ + private void initRefeshThread (boolean runImmediately) { if (refreshInterval > 0) { - refreshUsed = new Thread(new RefreshThread(this), + refreshUsed = new Thread(new RefreshThread(this, runImmediately), "refreshUsed-" + dirPath); refreshUsed.setDaemon(true); refreshUsed.start(); @@ -101,6 +116,14 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { protected abstract void refresh(); /** + * Reset that if we need to do the first refresh. + * @param shouldFirstRefresh The flag value to set. + */ + protected void setShouldFirstRefresh(boolean shouldFirstRefresh) { +this.shouldFirstRefresh = shouldFirstRefresh; + } + + /** * @return an estimate of space used in the directory path. */ @Override public long getUsed() throws IOException { @@ -156,9 +179,11 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { private static final class RefreshThread implements Runnable { final CachingGetSpaceUsed spaceUsed; +private boolean runImmediately; -RefreshThread(CachingGetSpaceUsed spaceUsed) { +RefreshThread(CachingGetSpaceUsed spaceUsed, boolean runImmediately) { this.spaceUsed = spaceUsed; + this.runImmediately = runImmediately; } @Override @@ -176,7 +201,10 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { } // Make sure that after the jitter we didn't end up at 0. refreshInterval = Math.max(refreshInterval, 1); - Thread.sleep(refreshInterval); + if (!runImmediately) { +Thread.sleep(refreshInterval); + } + runImmediately = false; // update the used variable spaceUsed.refresh(); } catch (InterruptedException e) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java index 3f73e01..7953d45 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java @@ -662,5 +662,11 @@ public interface
[hadoop] branch branch-2 updated: HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-2 by this push: new fc2cdb6 HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. fc2cdb6 is described below commit fc2cdb6cca8de95fb220c6a9f995406fa8b1d669 Author: Yiqun Lin AuthorDate: Thu Nov 28 10:43:35 2019 +0800 HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. (cherry picked from commit 2b452b4e6063072b2bec491edd3f412eb7ac21f3) --- .../org/apache/hadoop/fs/CachingGetSpaceUsed.java | 34 +++-- .../server/datanode/fsdataset/FsDatasetSpi.java| 6 +++ .../datanode/fsdataset/impl/FsDatasetImpl.java | 12 ++--- .../fsdataset/impl/ReplicaCachingGetSpaceUsed.java | 1 + .../impl/TestReplicaCachingGetSpaceUsed.java | 55 ++ 5 files changed, 98 insertions(+), 10 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java index 92476d7..58dc82d 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java @@ -47,6 +47,7 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { private final long jitter; private final String dirPath; private Thread refreshUsed; + private boolean shouldFirstRefresh; /** * This is the constructor used by the builder. @@ -79,16 +80,30 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { this.refreshInterval = interval; this.jitter = jitter; this.used.set(initialUsed); +this.shouldFirstRefresh = true; } void init() { if (used.get() < 0) { used.set(0); + if (!shouldFirstRefresh) { +// Skip initial refresh operation, so we need to do first refresh +// operation immediately in refresh thread. +initRefeshThread(true); +return; + } refresh(); } +initRefeshThread(false); + } + /** + * RunImmediately should set true, if we skip the first refresh. + * @param runImmediately The param default should be false. + */ + private void initRefeshThread (boolean runImmediately) { if (refreshInterval > 0) { - refreshUsed = new Thread(new RefreshThread(this), + refreshUsed = new Thread(new RefreshThread(this, runImmediately), "refreshUsed-" + dirPath); refreshUsed.setDaemon(true); refreshUsed.start(); @@ -101,6 +116,14 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { protected abstract void refresh(); /** + * Reset that if we need to do the first refresh. + * @param shouldFirstRefresh The flag value to set. + */ + protected void setShouldFirstRefresh(boolean shouldFirstRefresh) { +this.shouldFirstRefresh = shouldFirstRefresh; + } + + /** * @return an estimate of space used in the directory path. */ @Override public long getUsed() throws IOException { @@ -156,9 +179,11 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { private static final class RefreshThread implements Runnable { final CachingGetSpaceUsed spaceUsed; +private boolean runImmediately; -RefreshThread(CachingGetSpaceUsed spaceUsed) { +RefreshThread(CachingGetSpaceUsed spaceUsed, boolean runImmediately) { this.spaceUsed = spaceUsed; + this.runImmediately = runImmediately; } @Override @@ -176,7 +201,10 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { } // Make sure that after the jitter we didn't end up at 0. refreshInterval = Math.max(refreshInterval, 1); - Thread.sleep(refreshInterval); + if (!runImmediately) { +Thread.sleep(refreshInterval); + } + runImmediately = false; // update the used variable spaceUsed.refresh(); } catch (InterruptedException e) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java index 3f73e01..7953d45 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java @@ -662,5 +662,11 @@ public interface FsDatasetSpi e
[hadoop] branch trunk updated: HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 2b452b4 HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 2b452b4 is described below commit 2b452b4e6063072b2bec491edd3f412eb7ac21f3 Author: Yiqun Lin AuthorDate: Thu Nov 28 10:43:35 2019 +0800 HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. --- .../org/apache/hadoop/fs/CachingGetSpaceUsed.java | 34 +++-- .../server/datanode/fsdataset/FsDatasetSpi.java| 6 +++ .../datanode/fsdataset/impl/FsDatasetImpl.java | 12 ++--- .../fsdataset/impl/ReplicaCachingGetSpaceUsed.java | 1 + .../impl/TestReplicaCachingGetSpaceUsed.java | 55 ++ 5 files changed, 98 insertions(+), 10 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java index 92476d7..58dc82d 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java @@ -47,6 +47,7 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { private final long jitter; private final String dirPath; private Thread refreshUsed; + private boolean shouldFirstRefresh; /** * This is the constructor used by the builder. @@ -79,16 +80,30 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { this.refreshInterval = interval; this.jitter = jitter; this.used.set(initialUsed); +this.shouldFirstRefresh = true; } void init() { if (used.get() < 0) { used.set(0); + if (!shouldFirstRefresh) { +// Skip initial refresh operation, so we need to do first refresh +// operation immediately in refresh thread. +initRefeshThread(true); +return; + } refresh(); } +initRefeshThread(false); + } + /** + * RunImmediately should set true, if we skip the first refresh. + * @param runImmediately The param default should be false. + */ + private void initRefeshThread (boolean runImmediately) { if (refreshInterval > 0) { - refreshUsed = new Thread(new RefreshThread(this), + refreshUsed = new Thread(new RefreshThread(this, runImmediately), "refreshUsed-" + dirPath); refreshUsed.setDaemon(true); refreshUsed.start(); @@ -101,6 +116,14 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { protected abstract void refresh(); /** + * Reset that if we need to do the first refresh. + * @param shouldFirstRefresh The flag value to set. + */ + protected void setShouldFirstRefresh(boolean shouldFirstRefresh) { +this.shouldFirstRefresh = shouldFirstRefresh; + } + + /** * @return an estimate of space used in the directory path. */ @Override public long getUsed() throws IOException { @@ -156,9 +179,11 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { private static final class RefreshThread implements Runnable { final CachingGetSpaceUsed spaceUsed; +private boolean runImmediately; -RefreshThread(CachingGetSpaceUsed spaceUsed) { +RefreshThread(CachingGetSpaceUsed spaceUsed, boolean runImmediately) { this.spaceUsed = spaceUsed; + this.runImmediately = runImmediately; } @Override @@ -176,7 +201,10 @@ public abstract class CachingGetSpaceUsed implements Closeable, GetSpaceUsed { } // Make sure that after the jitter we didn't end up at 0. refreshInterval = Math.max(refreshInterval, 1); - Thread.sleep(refreshInterval); + if (!runImmediately) { +Thread.sleep(refreshInterval); + } + runImmediately = false; // update the used variable spaceUsed.refresh(); } catch (InterruptedException e) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java index 5b1e37a..e53cb37 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java @@ -661,5 +661,11 @@ public interface FsDatasetSpi extends FSDatasetMBean { */ AutoCloseableLock acquireDatasetLock(); + /*
[hadoop] branch trunk updated: HDFS-14649. Add suspect probe for DeadNodeDetector. Contributed by Lisheng Sun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new c8bef4d HDFS-14649. Add suspect probe for DeadNodeDetector. Contributed by Lisheng Sun. c8bef4d is described below commit c8bef4d6a6d7d5affd00cff6ea4a2e2ef778050e Author: Yiqun Lin AuthorDate: Wed Nov 27 10:57:20 2019 +0800 HDFS-14649. Add suspect probe for DeadNodeDetector. Contributed by Lisheng Sun. --- .../org/apache/hadoop/hdfs/DeadNodeDetector.java | 169 + .../hadoop/hdfs/client/HdfsClientConfigKeys.java | 13 ++ .../src/main/resources/hdfs-default.xml| 24 +++ .../apache/hadoop/hdfs/TestDeadNodeDetection.java | 104 +++-- 4 files changed, 264 insertions(+), 46 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java index 2fe7cf8..ce50547 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java @@ -17,6 +17,7 @@ */ package org.apache.hadoop.hdfs; +import com.google.common.annotations.VisibleForTesting; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; @@ -48,8 +49,14 @@ import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_DEAD_NODE_INTERVAL_MS_KEY; import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_DEAD_NODE_THREADS_DEFAULT; import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_DEAD_NODE_THREADS_KEY; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_SUSPECT_NODE_INTERVAL_MS_DEFAULT; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_SUSPECT_NODE_INTERVAL_MS_KEY; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_SUSPECT_NODE_THREADS_DEFAULT; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_SUSPECT_NODE_THREADS_KEY; import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_RPC_THREADS_DEFAULT; import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_RPC_THREADS_KEY; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_SUSPECT_NODE_QUEUE_MAX_DEFAULT; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_SUSPECT_NODE_QUEUE_MAX_KEY; import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_SOCKET_TIMEOUT_KEY; /** @@ -83,13 +90,13 @@ public class DeadNodeDetector implements Runnable { private final Map deadNodes; /** - * Record dead nodes by one DFSInputStream. When dead node is not used by one - * DFSInputStream, remove it from dfsInputStreamNodes#DFSInputStream. If - * DFSInputStream does not include any dead node, remove DFSInputStream from - * dfsInputStreamNodes. + * Record suspect and dead nodes by one DFSInputStream. When node is not used + * by one DFSInputStream, remove it from suspectAndDeadNodes#DFSInputStream. + * If DFSInputStream does not include any node, remove DFSInputStream from + * suspectAndDeadNodes. */ private final Map> - dfsInputStreamNodes; + suspectAndDeadNodes; /** * Datanodes that is being probed. @@ -108,11 +115,21 @@ public class DeadNodeDetector implements Runnable { private long deadNodeDetectInterval = 0; /** + * Interval time in milliseconds for probing suspect node behavior. + */ + private long suspectNodeDetectInterval = 0; + + /** * The max queue size of probing dead node. */ private int maxDeadNodesProbeQueueLen = 0; /** + * The max queue size of probing suspect node. + */ + private int maxSuspectNodesProbeQueueLen; + + /** * Connection timeout for probing dead node in milliseconds. */ private long probeConnectionTimeoutMs; @@ -123,16 +140,31 @@ public class DeadNodeDetector implements Runnable { private Queue deadNodesProbeQueue; /** + * The suspect node probe queue. + */ + private Queue suspectNodesProbeQueue; + + /** * The thread pool of probing dead node. */ private ExecutorService probeDeadNodesThreadPool; /** + * The thread pool of probing suspect n
[hadoop] branch trunk updated: HDFS-14651. DeadNodeDetector checks dead node periodically. Contributed by Lisheng Sun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 9b6906f HDFS-14651. DeadNodeDetector checks dead node periodically. Contributed by Lisheng Sun. 9b6906f is described below commit 9b6906fe914829f50076c2291dba59d425475d7b Author: Yiqun Lin AuthorDate: Fri Nov 22 10:53:55 2019 +0800 HDFS-14651. DeadNodeDetector checks dead node periodically. Contributed by Lisheng Sun. --- .../java/org/apache/hadoop/hdfs/ClientContext.java | 2 +- .../org/apache/hadoop/hdfs/DeadNodeDetector.java | 308 - .../hadoop/hdfs/client/HdfsClientConfigKeys.java | 22 ++ .../src/main/resources/hdfs-default.xml| 40 +++ .../apache/hadoop/hdfs/TestDeadNodeDetection.java | 154 +-- 5 files changed, 504 insertions(+), 22 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java index abb039c..6ee5277 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java @@ -150,7 +150,7 @@ public class ClientContext { conf.getWriteByteArrayManagerConf()); this.deadNodeDetectionEnabled = conf.isDeadNodeDetectionEnabled(); if (deadNodeDetectionEnabled && deadNodeDetector == null) { - deadNodeDetector = new DeadNodeDetector(name); + deadNodeDetector = new DeadNodeDetector(name, config); deadNodeDetectorThr = new Daemon(deadNodeDetector); deadNodeDetectorThr.start(); } diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java index 1ac29a7..2fe7cf8 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DeadNodeDetector.java @@ -17,13 +17,40 @@ */ package org.apache.hadoop.hdfs; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol; import org.apache.hadoop.hdfs.protocol.DatanodeInfo; +import org.apache.hadoop.hdfs.protocol.DatanodeLocalInfo; +import org.apache.hadoop.hdfs.protocol.HdfsConstants; +import org.apache.hadoop.util.Daemon; +import org.apache.hadoop.util.Time; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.util.HashSet; +import java.util.Map; +import java.util.Queue; import java.util.Set; +import java.util.concurrent.ArrayBlockingQueue; +import java.util.concurrent.Callable; import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; + +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_DEAD_NODE_QUEUE_MAX_DEFAULT; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_DEAD_NODE_QUEUE_MAX_KEY; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_CONNECTION_TIMEOUT_MS_DEFAULT; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_CONNECTION_TIMEOUT_MS_KEY; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_DEAD_NODE_INTERVAL_MS_DEFAULT; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_DEAD_NODE_INTERVAL_MS_KEY; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_DEAD_NODE_THREADS_DEFAULT; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_PROBE_DEAD_NODE_THREADS_KEY; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_RPC_THREADS_DEFAULT; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_DEAD_NODE_DETECTION_RPC_THREADS_KEY; +import static org.apache.hadoop.hdfs.client.HdfsClientConfigKeys.DFS_CLIENT_SOCKET_TIMEOUT_KEY; /** * Detect the dead nodes in advance, and share this information among all the @@ -48,10 +75,12 @@ public class DeadNodeDetector implements Runnable { */ private String name; + private Configuration conf; + /** * Dead nodes shared by all the DFSInputStreams of the client. */ - private final ConcurrentHashMap deadNodes; + private final Map d
[hadoop] branch trunk updated: HDFS-14648. Implement DeadNodeDetector basic model. Contributed by Lisheng Sun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new b3119b9 HDFS-14648. Implement DeadNodeDetector basic model. Contributed by Lisheng Sun. b3119b9 is described below commit b3119b9ab60a19d624db476c4e1c53410870c7a6 Author: Yiqun Lin AuthorDate: Sat Nov 16 11:32:41 2019 +0800 HDFS-14648. Implement DeadNodeDetector basic model. Contributed by Lisheng Sun. --- .../java/org/apache/hadoop/hdfs/ClientContext.java | 49 ++ .../java/org/apache/hadoop/hdfs/DFSClient.java | 98 +++ .../org/apache/hadoop/hdfs/DFSInputStream.java | 98 +++ .../apache/hadoop/hdfs/DFSStripedInputStream.java | 6 +- .../org/apache/hadoop/hdfs/DeadNodeDetector.java | 185 + .../hadoop/hdfs/client/HdfsClientConfigKeys.java | 4 + .../hadoop/hdfs/client/impl/DfsClientConf.java | 31 +++- .../src/main/resources/hdfs-default.xml| 9 + .../apache/hadoop/hdfs/TestDeadNodeDetection.java | 183 9 files changed, 617 insertions(+), 46 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java index ad1b359..abb039c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientContext.java @@ -40,6 +40,7 @@ import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.net.NetworkTopology; import org.apache.hadoop.net.NodeBase; import org.apache.hadoop.net.ScriptBasedMapping; +import org.apache.hadoop.util.Daemon; import org.apache.hadoop.util.ReflectionUtils; import com.google.common.annotations.VisibleForTesting; @@ -118,6 +119,19 @@ public class ClientContext { private NodeBase clientNode; private boolean topologyResolutionEnabled; + private Daemon deadNodeDetectorThr = null; + + /** + * The switch to DeadNodeDetector. + */ + private boolean deadNodeDetectionEnabled = false; + + /** + * Detect the dead datanodes in advance, and share this information among all + * the DFSInputStreams in the same client. + */ + private DeadNodeDetector deadNodeDetector = null; + private ClientContext(String name, DfsClientConf conf, Configuration config) { final ShortCircuitConf scConf = conf.getShortCircuitConf(); @@ -134,6 +148,12 @@ public class ClientContext { this.byteArrayManager = ByteArrayManager.newInstance( conf.getWriteByteArrayManagerConf()); +this.deadNodeDetectionEnabled = conf.isDeadNodeDetectionEnabled(); +if (deadNodeDetectionEnabled && deadNodeDetector == null) { + deadNodeDetector = new DeadNodeDetector(name); + deadNodeDetectorThr = new Daemon(deadNodeDetector); + deadNodeDetectorThr.start(); +} initTopologyResolution(config); } @@ -251,4 +271,33 @@ public class ClientContext { datanodeInfo.getNetworkLocation()); return NetworkTopology.getDistanceByPath(clientNode, node); } + + /** + * The switch to DeadNodeDetector. If true, DeadNodeDetector is available. + */ + public boolean isDeadNodeDetectionEnabled() { +return deadNodeDetectionEnabled; + } + + /** + * Obtain DeadNodeDetector of the current client. + */ + public DeadNodeDetector getDeadNodeDetector() { +return deadNodeDetector; + } + + /** + * Close dead node detector thread. + */ + public void stopDeadNodeDetectorThread() { +if (deadNodeDetectorThr != null) { + deadNodeDetectorThr.interrupt(); + try { +deadNodeDetectorThr.join(3000); + } catch (InterruptedException e) { +LOG.warn("Encountered exception while waiting to join on dead " + +"node detector thread.", e); + } +} + } } diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java index 56280f3..c19aa96 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java @@ -44,6 +44,8 @@ import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Random; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.SynchronousQueue; import java.util.concurrent.ThreadLocalRandom; import java.util.concurrent.ThreadPoolExecutor; @@ -631,6 +633,8 @@ public class DFSClient implements java.io.Closeable, RemotePeerFactory, // lease renewal sto
[hadoop] branch branch-3.0 updated: HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new ea53306 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. ea53306 is described below commit ea5330643ce353cc34ed81bb988cc744502a15e3 Author: Yiqun Lin AuthorDate: Wed Aug 7 23:05:09 2019 +0800 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. --- .../hadoop/fs/CommonConfigurationKeysPublic.java | 19 +++ .../java/org/apache/hadoop/fs/GetSpaceUsed.java| 31 +++-- .../src/main/resources/core-default.xml| 21 +++ .../server/datanode/FSCachingGetSpaceUsed.java | 82 .../server/datanode/fsdataset/FsDatasetSpi.java| 2 + .../datanode/fsdataset/impl/BlockPoolSlice.java| 11 +- .../datanode/fsdataset/impl/FsDatasetImpl.java | 13 ++ .../fsdataset/impl/ReplicaCachingGetSpaceUsed.java | 108 +++ .../hdfs/server/datanode/SimulatedFSDataset.java | 12 ++ .../datanode/extdataset/ExternalDatasetImpl.java | 6 + .../impl/TestReplicaCachingGetSpaceUsed.java | 148 + 11 files changed, 437 insertions(+), 16 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java index 95fccec..8d6a043 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java @@ -72,6 +72,25 @@ public class CommonConfigurationKeysPublic { public static final String FS_DU_INTERVAL_KEY = "fs.du.interval"; /** Default value for FS_DU_INTERVAL_KEY */ public static final longFS_DU_INTERVAL_DEFAULT = 60; + + /** + * @see + * + * core-default.xml + */ + public static final String FS_GETSPACEUSED_CLASSNAME = + "fs.getspaceused.classname"; + + /** + * @see + * + * core-default.xml + */ + public static final String FS_GETSPACEUSED_JITTER_KEY = + "fs.getspaceused.jitterMillis"; + /** Default value for FS_GETSPACEUSED_JITTER_KEY */ + public static final long FS_GETSPACEUSED_JITTER_DEFAULT = 6; + /** * @see * diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java index 4d1f9ef..3439317 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java @@ -26,7 +26,6 @@ import java.io.File; import java.io.IOException; import java.lang.reflect.Constructor; import java.lang.reflect.InvocationTargetException; -import java.util.concurrent.TimeUnit; public interface GetSpaceUsed { @@ -36,20 +35,16 @@ public interface GetSpaceUsed { /** * The builder class */ - final class Builder { + class Builder { static final Logger LOG = LoggerFactory.getLogger(Builder.class); -static final String CLASSNAME_KEY = "fs.getspaceused.classname"; -static final String JITTER_KEY = "fs.getspaceused.jitterMillis"; -static final long DEFAULT_JITTER = TimeUnit.MINUTES.toMillis(1); - - private Configuration conf; private Class klass = null; private File path = null; private Long interval = null; private Long jitter = null; private Long initialUsed = null; +private Constructor cons; public Configuration getConf() { return conf; @@ -89,7 +84,8 @@ public interface GetSpaceUsed { if (conf == null) { return result; } - return conf.getClass(CLASSNAME_KEY, result, GetSpaceUsed.class); + return conf.getClass(CommonConfigurationKeys.FS_GETSPACEUSED_CLASSNAME, + result, GetSpaceUsed.class); } public Builder setKlass(Class klass) { @@ -124,9 +120,10 @@ public interface GetSpaceUsed { Configuration configuration = this.conf; if (configuration == null) { - return DEFAULT_JITTER; + return CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_DEFAULT; } -return configuration.getLong(JITTER_KEY, DEFAULT_JITTER); +return configuration.getLong(CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_KEY, +CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_DEFAULT); } return jitter; } @@ -136,11 +133,21 @@ public interface GetSpaceUsed {
[hadoop] branch branch-2 updated: HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-2 by this push: new 63531be HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 63531be is described below commit 63531be5057f60c1b16d02b88058608470ed566f Author: Yiqun Lin AuthorDate: Wed Aug 7 23:01:51 2019 +0800 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. --- .../hadoop/fs/CommonConfigurationKeysPublic.java | 19 +++ .../java/org/apache/hadoop/fs/GetSpaceUsed.java| 31 +++-- .../src/main/resources/core-default.xml| 21 +++ .../server/datanode/FSCachingGetSpaceUsed.java | 82 .../server/datanode/fsdataset/FsDatasetSpi.java| 2 + .../datanode/fsdataset/impl/BlockPoolSlice.java| 11 +- .../datanode/fsdataset/impl/FsDatasetImpl.java | 13 ++ .../fsdataset/impl/ReplicaCachingGetSpaceUsed.java | 108 +++ .../hdfs/server/datanode/SimulatedFSDataset.java | 12 ++ .../datanode/extdataset/ExternalDatasetImpl.java | 6 + .../impl/TestReplicaCachingGetSpaceUsed.java | 148 + 11 files changed, 437 insertions(+), 16 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java index 9adb9ba..04bd936 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java @@ -84,6 +84,25 @@ public class CommonConfigurationKeysPublic { public static final String FS_DU_INTERVAL_KEY = "fs.du.interval"; /** Default value for FS_DU_INTERVAL_KEY */ public static final longFS_DU_INTERVAL_DEFAULT = 60; + + /** + * @see + * + * core-default.xml + */ + public static final String FS_GETSPACEUSED_CLASSNAME = + "fs.getspaceused.classname"; + + /** + * @see + * + * core-default.xml + */ + public static final String FS_GETSPACEUSED_JITTER_KEY = + "fs.getspaceused.jitterMillis"; + /** Default value for FS_GETSPACEUSED_JITTER_KEY */ + public static final long FS_GETSPACEUSED_JITTER_DEFAULT = 6; + /** * @see * diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java index 4d1f9ef..3439317 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java @@ -26,7 +26,6 @@ import java.io.File; import java.io.IOException; import java.lang.reflect.Constructor; import java.lang.reflect.InvocationTargetException; -import java.util.concurrent.TimeUnit; public interface GetSpaceUsed { @@ -36,20 +35,16 @@ public interface GetSpaceUsed { /** * The builder class */ - final class Builder { + class Builder { static final Logger LOG = LoggerFactory.getLogger(Builder.class); -static final String CLASSNAME_KEY = "fs.getspaceused.classname"; -static final String JITTER_KEY = "fs.getspaceused.jitterMillis"; -static final long DEFAULT_JITTER = TimeUnit.MINUTES.toMillis(1); - - private Configuration conf; private Class klass = null; private File path = null; private Long interval = null; private Long jitter = null; private Long initialUsed = null; +private Constructor cons; public Configuration getConf() { return conf; @@ -89,7 +84,8 @@ public interface GetSpaceUsed { if (conf == null) { return result; } - return conf.getClass(CLASSNAME_KEY, result, GetSpaceUsed.class); + return conf.getClass(CommonConfigurationKeys.FS_GETSPACEUSED_CLASSNAME, + result, GetSpaceUsed.class); } public Builder setKlass(Class klass) { @@ -124,9 +120,10 @@ public interface GetSpaceUsed { Configuration configuration = this.conf; if (configuration == null) { - return DEFAULT_JITTER; + return CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_DEFAULT; } -return configuration.getLong(JITTER_KEY, DEFAULT_JITTER); +return configuration.getLong(CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_KEY, +CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_DEFAULT); } return jitter; } @@ -136,11 +133,21 @@ public interface GetSpaceUsed { ret
[hadoop] branch trunk updated: HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new a5bb1e8 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. a5bb1e8 is described below commit a5bb1e8ee871dfff77d0f6921b13c8ffb50e Author: Yiqun Lin AuthorDate: Wed Aug 7 10:18:11 2019 +0800 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. --- .../hadoop/fs/CommonConfigurationKeysPublic.java | 19 +++ .../java/org/apache/hadoop/fs/GetSpaceUsed.java| 31 +++-- .../src/main/resources/core-default.xml| 20 +++ .../server/datanode/FSCachingGetSpaceUsed.java | 82 .../server/datanode/fsdataset/FsDatasetSpi.java| 2 + .../datanode/fsdataset/impl/BlockPoolSlice.java| 13 +- .../datanode/fsdataset/impl/FsDatasetImpl.java | 13 ++ .../fsdataset/impl/ReplicaCachingGetSpaceUsed.java | 108 +++ .../hdfs/server/datanode/SimulatedFSDataset.java | 11 ++ .../datanode/extdataset/ExternalDatasetImpl.java | 6 + .../impl/TestReplicaCachingGetSpaceUsed.java | 148 + 11 files changed, 437 insertions(+), 16 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java index 7e3e47c..b24ce9e 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java @@ -72,6 +72,25 @@ public class CommonConfigurationKeysPublic { public static final String FS_DU_INTERVAL_KEY = "fs.du.interval"; /** Default value for FS_DU_INTERVAL_KEY */ public static final longFS_DU_INTERVAL_DEFAULT = 60; + + /** + * @see + * + * core-default.xml + */ + public static final String FS_GETSPACEUSED_CLASSNAME = + "fs.getspaceused.classname"; + + /** + * @see + * + * core-default.xml + */ + public static final String FS_GETSPACEUSED_JITTER_KEY = + "fs.getspaceused.jitterMillis"; + /** Default value for FS_GETSPACEUSED_JITTER_KEY */ + public static final long FS_GETSPACEUSED_JITTER_DEFAULT = 6; + /** * @see * diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java index 4d1f9ef..3439317 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GetSpaceUsed.java @@ -26,7 +26,6 @@ import java.io.File; import java.io.IOException; import java.lang.reflect.Constructor; import java.lang.reflect.InvocationTargetException; -import java.util.concurrent.TimeUnit; public interface GetSpaceUsed { @@ -36,20 +35,16 @@ public interface GetSpaceUsed { /** * The builder class */ - final class Builder { + class Builder { static final Logger LOG = LoggerFactory.getLogger(Builder.class); -static final String CLASSNAME_KEY = "fs.getspaceused.classname"; -static final String JITTER_KEY = "fs.getspaceused.jitterMillis"; -static final long DEFAULT_JITTER = TimeUnit.MINUTES.toMillis(1); - - private Configuration conf; private Class klass = null; private File path = null; private Long interval = null; private Long jitter = null; private Long initialUsed = null; +private Constructor cons; public Configuration getConf() { return conf; @@ -89,7 +84,8 @@ public interface GetSpaceUsed { if (conf == null) { return result; } - return conf.getClass(CLASSNAME_KEY, result, GetSpaceUsed.class); + return conf.getClass(CommonConfigurationKeys.FS_GETSPACEUSED_CLASSNAME, + result, GetSpaceUsed.class); } public Builder setKlass(Class klass) { @@ -124,9 +120,10 @@ public interface GetSpaceUsed { Configuration configuration = this.conf; if (configuration == null) { - return DEFAULT_JITTER; + return CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_DEFAULT; } -return configuration.getLong(JITTER_KEY, DEFAULT_JITTER); +return configuration.getLong(CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_KEY, +CommonConfigurationKeys.FS_GETSPACEUSED_JITTER_DEFAULT); } return jitter; } @@ -136,11 +133,21 @@ public interface GetSpaceUsed { ret
[hadoop] branch trunk updated: HDFS-14410. Make Dynamometer documentation properly compile onto the Hadoop site. Contributed by Erik Krogen.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 5043840 HDFS-14410. Make Dynamometer documentation properly compile onto the Hadoop site. Contributed by Erik Krogen. 5043840 is described below commit 5043840b1daeb80c1e3180ba7df604830cf69ea4 Author: Yiqun Lin AuthorDate: Thu Jul 11 23:47:27 2019 +0800 HDFS-14410. Make Dynamometer documentation properly compile onto the Hadoop site. Contributed by Erik Krogen. --- hadoop-project/src/site/site.xml | 1 + .../src/site/markdown/Dynamometer.md | 299 + .../src/site/resources/css/site.css| 30 +++ .../images/dynamometer-architecture-infra.png | Bin 0 -> 123874 bytes .../images/dynamometer-architecture-replay.png | Bin 0 -> 159507 bytes 5 files changed, 330 insertions(+) diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index bf6bd26..a790369 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -215,6 +215,7 @@ + diff --git a/hadoop-tools/hadoop-dynamometer/src/site/markdown/Dynamometer.md b/hadoop-tools/hadoop-dynamometer/src/site/markdown/Dynamometer.md new file mode 100644 index 000..39dd0db --- /dev/null +++ b/hadoop-tools/hadoop-dynamometer/src/site/markdown/Dynamometer.md @@ -0,0 +1,299 @@ + + +# Dynamometer Guide + + + +## Overview + +Dynamometer is a tool to performance test Hadoop's HDFS NameNode. The intent is to provide a +real-world environment by initializing the NameNode against a production file system image and replaying +a production workload collected via e.g. the NameNode's audit logs. This allows for replaying a workload +which is not only similar in characteristic to that experienced in production, but actually identical. + +Dynamometer will launch a YARN application which starts a single NameNode and a configurable number of +DataNodes, simulating an entire HDFS cluster as a single application. There is an additional `workload` +job run as a MapReduce job which accepts audit logs as input and uses the information contained within to +submit matching requests to the NameNode, inducing load on the service. + +Dynamometer can execute this same workload against different Hadoop versions or with different +configurations, allowing for the testing of configuration tweaks and code changes at scale without the +necessity of deploying to a real large-scale cluster. + +Throughout this documentation, we will use "Dyno-HDFS", "Dyno-NN", and "Dyno-DN" to refer to the HDFS +cluster, NameNode, and DataNodes (respectively) which are started _inside of_ a Dynamometer application. +Terms like HDFS, YARN, and NameNode used without qualification refer to the existing infrastructure on +top of which Dynamometer is run. + +For more details on how Dynamometer works, as opposed to how to use it, see the Architecture section +at the end of this page. + +## Requirements + +Dynamometer is based around YARN applications, so an existing YARN cluster will be required for execution. +It also requires an accompanying HDFS instance to store some temporary files for communication. + +## Building + +Dynamometer consists of three main components, each one in its own module: + +* Infrastructure (`dynamometer-infra`): This is the YARN application which starts a Dyno-HDFS cluster. +* Workload (`dynamometer-workload`): This is the MapReduce job which replays audit logs. +* Block Generator (`dynamometer-blockgen`): This is a MapReduce job used to generate input files for each Dyno-DN; its + execution is a prerequisite step to running the infrastructure application. + +The compiled version of all of these components will be included in a standard Hadoop distribution. +You can find them in the packaged distribution within `share/hadoop/tools/dynamometer`. + +## Setup Steps + +Before launching a Dynamometer application, there are a number of setup steps that must be completed, +instructing Dynamometer what configurations to use, what version to use, what fsimage to use when +loading, etc. These steps can be performed a single time to put everything in place, and then many +Dynamometer executions can be performed against them with minor tweaks to measure variations. + +Scripts discussed below can be found in the `share/hadoop/tools/dynamometer/dynamometer-{infra,workload,blockgen}/bin` +directories of the distribution. The corresponding Java JAR files can be found in the `share/hadoop/tools/lib/` directory. +References to bin files below assume that the current working directory is `share/hadoop/tools/dynamometer`. + +### Step 1: Preparing Requisite Files + +A number of steps are required in advance of startin
[hadoop] branch branch-2 updated: HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch branch-2 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/branch-2 by this push: new 78aebee HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 78aebee is described below commit 78aebee5c5f2cf8260137f29ff9be05f0e5b33b9 Author: Yiqun Lin AuthorDate: Wed Jul 10 10:53:34 2019 +0800 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. (cherry picked from commit 993dc8726b7d40ac832ae3e23b64e8541b62c4bd) --- .../blockmanagement/BlockManagerSafeMode.java | 22 +--- .../java/org/apache/hadoop/hdfs/TestSafeMode.java | 10 .../blockmanagement/TestBlockManagerSafeMode.java | 25 +++ .../hdfs/server/namenode/ha/TestHASafeMode.java| 29 ++ 4 files changed, 62 insertions(+), 24 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java index ae2f942..5271004 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java @@ -306,16 +306,20 @@ class BlockManagerSafeMode { } } -int numLive = blockManager.getDatanodeManager().getNumLiveDataNodes(); -if (numLive < datanodeThreshold) { - msg += String.format( - "The number of live datanodes %d needs an additional %d live " - + "datanodes to reach the minimum number %d.%n", - numLive, (datanodeThreshold - numLive), datanodeThreshold); +if (datanodeThreshold > 0) { + int numLive = blockManager.getDatanodeManager().getNumLiveDataNodes(); + if (numLive < datanodeThreshold) { +msg += String.format( +"The number of live datanodes %d needs an additional %d live " ++ "datanodes to reach the minimum number %d.%n", +numLive, (datanodeThreshold - numLive), datanodeThreshold); + } else { +msg += String.format("The number of live datanodes %d has reached " ++ "the minimum number %d. ", +numLive, datanodeThreshold); + } } else { - msg += String.format("The number of live datanodes %d has reached " - + "the minimum number %d. ", - numLive, datanodeThreshold); + msg += "The minimum number of live datanodes is not required. "; } if (getBytesInFuture() > 0) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java index 08926f3..8325a28 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java @@ -199,11 +199,11 @@ public class TestSafeMode { final NameNode nn = cluster.getNameNode(); String status = nn.getNamesystem().getSafemode(); -assertEquals("Safe mode is ON. The reported blocks 0 needs additional " + -"14 blocks to reach the threshold 0.9990 of total blocks 15." + NEWLINE + -"The number of live datanodes 0 has reached the minimum number 0. " + -"Safe mode will be turned off automatically once the thresholds " + -"have been reached.", status); +assertEquals("Safe mode is ON. The reported blocks 0 needs additional " ++ "14 blocks to reach the threshold 0.9990 of total blocks 15." ++ NEWLINE + "The minimum number of live datanodes is not required. " ++ "Safe mode will be turned off automatically once the thresholds have " ++ "been reached.", status); assertFalse("Mis-replicated block queues should not be initialized " + "until threshold is crossed", NameNodeAdapter.safeModeInitializedReplQueues(nn)); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java index b7e8f6d..eb58f5c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java +++ b/hadoop-hdfs
[hadoop] branch trunk updated: HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 993dc87 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 993dc87 is described below commit 993dc8726b7d40ac832ae3e23b64e8541b62c4bd Author: Yiqun Lin AuthorDate: Wed Jul 10 10:53:34 2019 +0800 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. --- .../blockmanagement/BlockManagerSafeMode.java | 22 +--- .../java/org/apache/hadoop/hdfs/TestSafeMode.java | 10 .../blockmanagement/TestBlockManagerSafeMode.java | 25 +++ .../hdfs/server/namenode/ha/TestHASafeMode.java| 29 ++ 4 files changed, 62 insertions(+), 24 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java index 1d7e69b..3f59b64 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerSafeMode.java @@ -309,16 +309,20 @@ class BlockManagerSafeMode { } } -int numLive = blockManager.getDatanodeManager().getNumLiveDataNodes(); -if (numLive < datanodeThreshold) { - msg += String.format( - "The number of live datanodes %d needs an additional %d live " - + "datanodes to reach the minimum number %d.%n", - numLive, (datanodeThreshold - numLive), datanodeThreshold); +if (datanodeThreshold > 0) { + int numLive = blockManager.getDatanodeManager().getNumLiveDataNodes(); + if (numLive < datanodeThreshold) { +msg += String.format( +"The number of live datanodes %d needs an additional %d live " ++ "datanodes to reach the minimum number %d.%n", +numLive, (datanodeThreshold - numLive), datanodeThreshold); + } else { +msg += String.format("The number of live datanodes %d has reached " ++ "the minimum number %d. ", +numLive, datanodeThreshold); + } } else { - msg += String.format("The number of live datanodes %d has reached " - + "the minimum number %d. ", - numLive, datanodeThreshold); + msg += "The minimum number of live datanodes is not required. "; } if (getBytesInFuture() > 0) { diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java index 0fde81e..7fbf222 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java @@ -201,11 +201,11 @@ public class TestSafeMode { final NameNode nn = cluster.getNameNode(); String status = nn.getNamesystem().getSafemode(); -assertEquals("Safe mode is ON. The reported blocks 0 needs additional " + -"14 blocks to reach the threshold 0.9990 of total blocks 15." + NEWLINE + -"The number of live datanodes 0 has reached the minimum number 0. " + -"Safe mode will be turned off automatically once the thresholds " + -"have been reached.", status); +assertEquals("Safe mode is ON. The reported blocks 0 needs additional " ++ "14 blocks to reach the threshold 0.9990 of total blocks 15." ++ NEWLINE + "The minimum number of live datanodes is not required. " ++ "Safe mode will be turned off automatically once the thresholds have " ++ "been reached.", status); assertFalse("Mis-replicated block queues should not be initialized " + "until threshold is crossed", NameNodeAdapter.safeModeInitializedReplQueues(nn)); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java index fd224ea..5cf094d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmana
[hadoop] branch trunk updated: HDDS-1189. Recon Aggregate DB schema and ORM. Contributed by Siddharth Wagle.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new a92806d HDDS-1189. Recon Aggregate DB schema and ORM. Contributed by Siddharth Wagle. a92806d is described below commit a92806d05a2eb1f586463fa07aa2f17ce9180401 Author: Yiqun Lin AuthorDate: Thu Apr 4 17:33:37 2019 +0800 HDDS-1189. Recon Aggregate DB schema and ORM. Contributed by Siddharth Wagle. --- .../common/src/main/resources/ozone-default.xml| 95 hadoop-hdds/tools/pom.xml | 3 +- hadoop-ozone/ozone-recon-codegen/pom.xml | 58 +++ .../ozone/recon/codegen/JooqCodeGenerator.java | 170 + .../recon/codegen/ReconSchemaGenerationModule.java | 39 + .../ozone/recon/codegen/TableNamingStrategy.java | 48 ++ .../hadoop/ozone/recon/codegen/package-info.java | 22 +++ .../ozone/recon/schema/ReconSchemaDefinition.java | 34 + .../recon/schema/UtilizationSchemaDefinition.java | 69 + .../hadoop/ozone/recon/schema/package-info.java| 22 +++ .../dev-support/findbugsExcludeFile.xml| 28 hadoop-ozone/ozone-recon/pom.xml | 147 ++ .../hadoop/ozone/recon/ReconControllerModule.java | 102 - .../hadoop/ozone/recon/ReconServerConfigKeys.java | 25 +++ .../recon/persistence/DataSourceConfiguration.java | 86 +++ .../persistence/DefaultDataSourceProvider.java | 74 + .../recon/persistence/JooqPersistenceModule.java | 111 ++ .../TransactionalMethodInterceptor.java| 76 + .../ozone/recon/persistence/package-info.java | 22 +++ .../recon/persistence/AbstractSqlDatabaseTest.java | 146 ++ .../TestUtilizationSchemaDefinition.java | 160 +++ .../ozone/recon/persistence/package-info.java | 22 +++ hadoop-ozone/pom.xml | 1 + 23 files changed, 1524 insertions(+), 36 deletions(-) diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml b/hadoop-hdds/common/src/main/resources/ozone-default.xml index 5580548..731bf28 100644 --- a/hadoop-hdds/common/src/main/resources/ozone-default.xml +++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml @@ -2394,4 +2394,99 @@ If enabled, tracing information is sent to tracing server. + +ozone.recon.sql.db.driver +org.sqlite.JDBC +OZONE, RECON + + Database driver class name available on the + Ozone Recon classpath. + + + +ozone.recon.sql.db.jdbc.url +jdbc:sqlite:/${ozone.recon.db.dir}/ozone_recon_sqlite.db +OZONE, RECON + + Ozone Recon SQL database jdbc url. + + + +ozone.recon.sql.db.username + +OZONE, RECON + + Ozone Recon SQL database username. + + + +ozone.recon.sql.db.password + +OZONE, RECON + + Ozone Recon datbase password. + + + +ozone.recon.sql.db.auto.commit +false +OZONE, RECON + + Sets the Ozone Recon database connection property of auto-commit to + true/false. + + + +ozone.recon.sql.db.conn.timeout +3 +OZONE, RECON + + Sets time in milliseconds before call to getConnection is timed out. + + + +ozone.recon.sql.db.conn.max.active +1 +OZONE, RECON + + The max active connections to the SQL database. The default SQLite + database only allows single active connection, set this to a + resonable value like 10, for external production database. + + + +ozone.recon.sql.db.conn.max.age +1800 +OZONE, RECON + + Sets maximum time a connection can be active in seconds. + + + +ozone.recon.sql.db.conn.idle.max.age +3600 +OZONE, RECON + + Sets maximum time to live for idle connection in seconds. + + + +ozone.recon.sql.db.conn.idle.test.period +60 +OZONE, RECON + + This sets the time (in seconds), for a connection to remain idle before + sending a test query to the DB. This is useful to prevent a DB from + timing out connections on its end. + + + +ozone.recon.sql.db.conn.idle.test +SELECT 1 +OZONE, RECON + + The query to send to the DB to maintain keep-alives and test for dead + connections. + + diff --git a/hadoop-hdds/tools/pom.xml b/hadoop-hdds/tools/pom.xml index 0e39330..689bca7 100644 --- a/hadoop-hdds/tools/pom.xml +++ b/hadoop-hdds/tools/pom.xml @@ -49,9 +49,8 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd";> org.xerial sqlite-jdbc - 3.8.7 + 3.25.2 - diff --git a/hadoop-ozone/ozone-recon-codegen/pom.xml b/hadoop-ozone/ozone-recon-codegen/pom.xml new file mode 100644 index
[hadoop] branch trunk updated: HDDS-1365. Fix error handling in KeyValueContainerCheck. Contributed by Supratim Deka.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new f96fb05 HDDS-1365. Fix error handling in KeyValueContainerCheck. Contributed by Supratim Deka. f96fb05 is described below commit f96fb05a2b889f3acdfae60d8f64c755ff48b8c1 Author: Yiqun Lin AuthorDate: Wed Apr 3 14:01:30 2019 +0800 HDDS-1365. Fix error handling in KeyValueContainerCheck. Contributed by Supratim Deka. --- .../container/keyvalue/KeyValueContainerCheck.java | 285 ++--- .../keyvalue/TestKeyValueContainerCheck.java | 11 +- 2 files changed, 81 insertions(+), 215 deletions(-) diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java index b1ab1e1..bdfdf21 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java @@ -68,9 +68,26 @@ public class KeyValueContainerCheck { } /** - * fast checks are basic and do not look inside the metadata files. - * Or into the structures on disk. These checks can be done on Open - * containers as well without concurrency implications + * Run basic integrity checks on container metadata. + * These checks do not look inside the metadata files. + * Applicable for OPEN containers. + * + * @return true : corruption detected, false : no corruption. + */ + public boolean fastCheck() { +boolean corruption = false; +try { + basicChecks(); + +} catch (IOException e) { + handleCorruption(e); + corruption = true; +} + +return corruption; + } + + /** * Checks : * 1. check directory layout * 2. check container file @@ -78,24 +95,14 @@ public class KeyValueContainerCheck { * @return void */ - public KvCheckError fastCheck() { - -KvCheckError error; -LOG.trace("Running fast check for container {};", containerID); - -error = loadContainerData(); -if (error != KvCheckError.ERROR_NONE) { - return error; -} + private void basicChecks() throws IOException { -error = checkLayout(); -if (error != KvCheckError.ERROR_NONE) { - return error; -} +LOG.trace("Running basic checks for container {};", containerID); -error = checkContainerFile(); +loadContainerData(); -return error; +checkLayout(); +checkContainerFile(); } /** @@ -107,129 +114,80 @@ public class KeyValueContainerCheck { * * fullCheck is a superset of fastCheck * - * @return void + * @return true : corruption detected, false : no corruption. */ - public KvCheckError fullCheck() { -/** - - */ -KvCheckError error; + public boolean fullCheck() { +boolean corruption = false; -error = fastCheck(); -if (error != KvCheckError.ERROR_NONE) { +try { + basicChecks(); + checkBlockDB(); - LOG.trace("fastCheck failed, aborting full check for Container {}", - containerID); - return error; +} catch (IOException e) { + handleCorruption(e); + corruption = true; } -error = checkBlockDB(); - -return error; +return corruption; } /** * Check the integrity of the directory structure of the container. - * - * @return error code or ERROR_NONE */ - private KvCheckError checkLayout() { -boolean success; -KvCheckError error = KvCheckError.ERROR_NONE; + private void checkLayout() throws IOException { // is metadataPath accessible as a directory? -try { - checkDirPath(metadataPath); -} catch (IOException ie) { - error = KvCheckError.METADATA_PATH_ACCESS; - handleCorruption(ie.getMessage(), error, ie); - return error; -} +checkDirPath(metadataPath); -String chunksPath = onDiskContainerData.getChunksPath(); // is chunksPath accessible as a directory? -try { - checkDirPath(chunksPath); -} catch (IOException ie) { - error = KvCheckError.CHUNKS_PATH_ACCESS; - handleCorruption(ie.getMessage(), error, ie); - return error; -} - -return error; +String chunksPath = onDiskContainerData.getChunksPath(); +checkDirPath(chunksPath); } private void checkDirPath(String path) throws IOException { File dirPath = new File(path); String errStr = null; -boolean success = true; try { if (!dirPath.isDirectory()) { -success = false; errStr = "Not a directory [" + path + "]"; +throw new IOException(errStr); }
[hadoop] branch trunk updated: HDDS-1337. Handle GroupMismatchException in OzoneClient. Contributed by Shashikant Banerjee.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new d31c868 HDDS-1337. Handle GroupMismatchException in OzoneClient. Contributed by Shashikant Banerjee. d31c868 is described below commit d31c86892e0ceec5d642f76fc9123fac4fd80db8 Author: Yiqun Lin AuthorDate: Tue Apr 2 16:27:11 2019 +0800 HDDS-1337. Handle GroupMismatchException in OzoneClient. Contributed by Shashikant Banerjee. --- .../hadoop/hdds/scm/storage/BlockOutputStream.java | 43 +++-- .../transport/server/ratis/XceiverServerRatis.java | 11 ++ hadoop-hdds/pom.xml| 2 +- .../hadoop/ozone/client/OzoneClientUtils.java | 2 + .../hadoop/ozone/client/io/KeyOutputStream.java| 15 +- .../org/apache/hadoop/ozone/MiniOzoneCluster.java | 2 + .../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 3 +- .../rpc/TestBlockOutputStreamWithFailures.java | 11 +- .../client/rpc/TestContainerStateMachine.java | 8 +- .../rpc/TestContainerStateMachineFailures.java | 8 +- .../rpc/TestOzoneClientRetriesOnException.java | 213 + .../ozone/container/ContainerTestHelper.java | 82 +++- hadoop-ozone/pom.xml | 2 +- 13 files changed, 359 insertions(+), 43 deletions(-) diff --git a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java index cfbb6ae..a8ead77 100644 --- a/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java +++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java @@ -55,6 +55,7 @@ import java.util.concurrent.TimeoutException; import java.util.concurrent.CompletionException; import java.util.concurrent.ExecutionException; import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicReference; import java.util.stream.Collectors; import static org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls @@ -100,7 +101,7 @@ public class BlockOutputStream extends OutputStream { // The IOException will be set by response handling thread in case there is an // exception received in the response. If the exception is set, the next // request will fail upfront. - private IOException ioException; + private AtomicReference ioException; private ExecutorService responseExecutor; // the effective length of data flushed so far @@ -187,6 +188,7 @@ public class BlockOutputStream extends OutputStream { writtenDataLength = 0; failedServers = Collections.emptyList(); bufferList = null; +ioException = new AtomicReference<>(null); } @@ -221,9 +223,8 @@ public class BlockOutputStream extends OutputStream { return bufferPool; } - @VisibleForTesting public IOException getIoException() { -return ioException; +return ioException.get(); } @VisibleForTesting @@ -372,10 +373,9 @@ public class BlockOutputStream extends OutputStream { waitOnFlushFutures(); } } catch (InterruptedException | ExecutionException e) { - ioException = new IOException( - "Unexpected Storage Container Exception: " + e.toString(), e); + setIoException(e); adjustBuffersOnException(); - throw ioException; + throw getIoException(); } if (!commitIndex2flushedDataMap.isEmpty()) { watchForCommit( @@ -430,9 +430,9 @@ public class BlockOutputStream extends OutputStream { adjustBuffers(index); } catch (TimeoutException | InterruptedException | ExecutionException e) { LOG.warn("watchForCommit failed for index " + commitIndex, e); + setIoException(e); adjustBuffersOnException(); - throw new IOException( - "Unexpected Storage Container Exception: " + e.toString(), e); + throw getIoException(); } } @@ -461,7 +461,7 @@ public class BlockOutputStream extends OutputStream { throw new CompletionException(sce); } // if the ioException is not set, putBlock is successful -if (ioException == null) { +if (getIoException() == null) { BlockID responseBlockID = BlockID.getFromProtobuf( e.getPutBlock().getCommittedBlockLength().getBlockID()); Preconditions.checkState(blockID.getContainerBlockID() @@ -505,10 +505,9 @@ public class BlockOutputStream extends OutputStream { } catch (InterruptedException | ExecutionException e) { // just set the exception here as well in order to maintain sanctity of // ioException field -ioException = new IOException( -"Unexpected Storage Container Exception: "
[hadoop] branch trunk updated: HDDS-1360. Invalid metric type due to fully qualified class name. Contributed by Doroszlai, Attila.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 8bfef21 HDDS-1360. Invalid metric type due to fully qualified class name. Contributed by Doroszlai, Attila. 8bfef21 is described below commit 8bfef21efaa8698976e6d80359c66116d7b451ea Author: Yiqun Lin AuthorDate: Mon Apr 1 19:26:44 2019 +0800 HDDS-1360. Invalid metric type due to fully qualified class name. Contributed by Doroszlai, Attila. --- .../java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java| 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java index 7466142..5e8e137 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java @@ -42,7 +42,8 @@ import org.apache.hadoop.metrics2.lib.Interns; public class SCMContainerMetrics implements MetricsSource { private final SCMMXBean scmmxBean; - private static final String SOURCE = SCMContainerMetrics.class.getName(); + private static final String SOURCE = + SCMContainerMetrics.class.getSimpleName(); public SCMContainerMetrics(SCMMXBean scmmxBean) { this.scmmxBean = scmmxBean; - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDDS-1334. Fix asf license errors in newly added files by HDDS-1234. Contributed by Aravindan Vijayan.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new c99b107 HDDS-1334. Fix asf license errors in newly added files by HDDS-1234. Contributed by Aravindan Vijayan. c99b107 is described below commit c99b107772f4a52832bafd3a4c23fdef8015fdea Author: Yiqun Lin AuthorDate: Tue Mar 26 11:51:04 2019 +0800 HDDS-1334. Fix asf license errors in newly added files by HDDS-1234. Contributed by Aravindan Vijayan. --- .../ozone/recon/AbstractOMMetadataManagerTest.java | 18 ++ 1 file changed, 18 insertions(+) diff --git a/hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/AbstractOMMetadataManagerTest.java b/hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/AbstractOMMetadataManagerTest.java index b58e225..d115891 100644 --- a/hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/AbstractOMMetadataManagerTest.java +++ b/hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/AbstractOMMetadataManagerTest.java @@ -1,3 +1,21 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + package org.apache.hadoop.ozone.recon; import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS; - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDDS-1234. Iterate the OM DB snapshot and populate the recon container DB. Contributed by Aravindan Vijayan.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new e5d72f5 HDDS-1234. Iterate the OM DB snapshot and populate the recon container DB. Contributed by Aravindan Vijayan. e5d72f5 is described below commit e5d72f504e2cf932657f96797623f3a5bbd71f4b Author: Yiqun Lin AuthorDate: Mon Mar 25 22:52:02 2019 +0800 HDDS-1234. Iterate the OM DB snapshot and populate the recon container DB. Contributed by Aravindan Vijayan. --- .../apache/hadoop/utils/LevelDBStoreIterator.java | 4 - .../org/apache/hadoop/utils/MetaStoreIterator.java | 5 - .../apache/hadoop/utils/RocksDBStoreIterator.java | 5 - .../java/org/apache/hadoop/utils/db/DBStore.java | 6 + .../org/apache/hadoop/utils/db/IntegerCodec.java | 28 ++- .../java/org/apache/hadoop/utils/db/RDBStore.java | 4 +- .../org/apache/hadoop/utils/TestMetadataStore.java | 51 - .../apache/hadoop/ozone/recon/ReconConstants.java | 4 + .../hadoop/ozone/recon/ReconControllerModule.java | 6 +- .../org/apache/hadoop/ozone/recon/ReconServer.java | 48 - .../ozone/recon/api/ContainerKeyService.java | 77 +++- .../ozone/recon/api/types/ContainerKeyPrefix.java | 41 +++- .../hadoop/ozone/recon/api/types/KeyMetadata.java | 74 --- .../recon/recovery/ReconOmMetadataManagerImpl.java | 4 +- .../recon/spi/ContainerDBServiceProvider.java | 13 +- .../recon/spi/OzoneManagerServiceProvider.java | 9 +- .../spi/impl/ContainerDBServiceProviderImpl.java | 116 ++- .../recon/spi/impl/ContainerKeyPrefixCodec.java| 87 + .../spi/impl/OzoneManagerServiceProviderImpl.java | 49 ++--- .../spi/{ => impl}/ReconContainerDBProvider.java | 62 +++--- .../ozone/recon/tasks/ContainerKeyMapperTask.java | 107 ++ .../package-info.java} | 21 +- .../ozone/recon/AbstractOMMetadataManagerTest.java | 172 .../apache/hadoop/ozone/recon/TestReconCodecs.java | 58 ++ .../apache/hadoop/ozone/recon/TestReconUtils.java | 4 +- .../ozone/recon/api/TestContainerKeyService.java | 216 + .../hadoop/ozone/recon/api/package-info.java} | 20 +- .../impl/TestContainerDBServiceProviderImpl.java | 141 ++ .../impl/TestOzoneManagerServiceProviderImpl.java | 111 +-- .../spi/impl/TestReconContainerDBProvider.java | 87 + .../recon/tasks/TestContainerKeyMapperTask.java| 194 ++ .../hadoop/ozone/recon/tasks/package-info.java}| 21 +- 32 files changed, 1416 insertions(+), 429 deletions(-) diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStoreIterator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStoreIterator.java index 92051dd..cd07b64 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStoreIterator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStoreIterator.java @@ -62,8 +62,4 @@ public class LevelDBStoreIterator implements MetaStoreIterator { levelDBIterator.seekToLast(); } - @Override - public void prefixSeek(byte[] prefix) { -levelDBIterator.seek(prefix); - } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetaStoreIterator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetaStoreIterator.java index 15ded0d..52d0a3e 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetaStoreIterator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetaStoreIterator.java @@ -36,9 +36,4 @@ public interface MetaStoreIterator extends Iterator { */ void seekToLast(); - /** - * seek with prefix. - */ - void prefixSeek(byte[] prefix); - } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreIterator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreIterator.java index 161d5de..6e9b695 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreIterator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreIterator.java @@ -63,9 +63,4 @@ public class RocksDBStoreIterator implements MetaStoreIterator { rocksDBIterator.seekToLast(); } - @Override - public void prefixSeek(byte[] prefix) { -rocksDBIterator.seek(prefix); - } - } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java index d55daa2..0bc30d0 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java @@ -19,6 +19,7 @@ package org.apache.hadoop.utils.db; +import java.io.F
[hadoop] branch trunk updated: HDDS-1233. Create an Ozone Manager Service provider for Recon. Contributed by Aravindan Vijayan.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 60cdd4c HDDS-1233. Create an Ozone Manager Service provider for Recon. Contributed by Aravindan Vijayan. 60cdd4c is described below commit 60cdd4cac17e547edbd9cd58c19ef27a8409b9c3 Author: Yiqun Lin AuthorDate: Thu Mar 21 11:02:29 2019 +0800 HDDS-1233. Create an Ozone Manager Service provider for Recon. Contributed by Aravindan Vijayan. --- .../org/apache/hadoop/utils/db/DBStoreBuilder.java | 8 +- .../hadoop/utils/db/RDBCheckpointManager.java | 46 .../java/org/apache/hadoop/utils/db/RDBStore.java | 25 +- .../apache/hadoop/utils/db/RocksDBCheckpoint.java | 81 ++ .../common/src/main/resources/ozone-default.xml| 85 ++- .../org/apache/hadoop/utils/db/TestRDBStore.java | 43 .../org/apache/hadoop/hdds/server/ServerUtils.java | 19 +- .../hadoop/ozone/om/OmMetadataManagerImpl.java | 145 ++- hadoop-ozone/ozone-recon/pom.xml | 26 +- .../apache/hadoop/ozone/recon/ReconConstants.java | 3 + .../hadoop/ozone/recon/ReconControllerModule.java | 14 +- .../hadoop/ozone/recon/ReconServerConfigKeys.java | 37 ++- .../org/apache/hadoop/ozone/recon/ReconUtils.java | 178 + .../ReconOMMetadataManager.java} | 24 +- .../recon/recovery/ReconOmMetadataManagerImpl.java | 99 .../recon/spi/OzoneManagerServiceProvider.java | 15 ++ .../ozone/recon/spi/ReconContainerDBProvider.java | 35 +-- .../spi/impl/OzoneManagerServiceProviderImpl.java | 211 .../apache/hadoop/ozone/recon/TestReconUtils.java | 135 ++ .../recovery/TestReconOmMetadataManagerImpl.java | 148 +++ .../hadoop/ozone/recon/recovery/package-info.java} | 8 +- .../impl/TestContainerDBServiceProviderImpl.java | 3 - .../impl/TestOzoneManagerServiceProviderImpl.java | 275 + 23 files changed, 1505 insertions(+), 158 deletions(-) diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java index 3459b20..34bdc5d 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStoreBuilder.java @@ -57,6 +57,7 @@ public final class DBStoreBuilder { private List tableNames; private Configuration configuration; private CodecRegistry registry; + private boolean readOnly = false; private DBStoreBuilder(Configuration configuration) { tables = new HashSet<>(); @@ -113,6 +114,11 @@ public final class DBStoreBuilder { return this; } + public DBStoreBuilder setReadOnly(boolean rdOnly) { +readOnly = rdOnly; +return this; + } + /** * Builds a DBStore instance and returns that. * @@ -131,7 +137,7 @@ public final class DBStoreBuilder { if (!dbFile.getParentFile().exists()) { throw new IOException("The DB destination directory should exist."); } -return new RDBStore(dbFile, options, tables, registry); +return new RDBStore(dbFile, options, tables, registry, readOnly); } /** diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBCheckpointManager.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBCheckpointManager.java index ce716c3..68d196f 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBCheckpointManager.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/RDBCheckpointManager.java @@ -19,13 +19,11 @@ package org.apache.hadoop.utils.db; -import java.io.IOException; import java.nio.file.Path; import java.nio.file.Paths; import java.time.Duration; import java.time.Instant; -import org.apache.commons.io.FileUtils; import org.apache.commons.lang3.StringUtils; import org.rocksdb.Checkpoint; import org.rocksdb.RocksDB; @@ -99,48 +97,4 @@ public class RDBCheckpointManager { } return null; } - - static class RocksDBCheckpoint implements DBCheckpoint { - -private Path checkpointLocation; -private long checkpointTimestamp; -private long latestSequenceNumber; -private long checkpointCreationTimeTaken; - -RocksDBCheckpoint(Path checkpointLocation, - long snapshotTimestamp, - long latestSequenceNumber, - long checkpointCreationTimeTaken) { - this.checkpointLocation = checkpointLocation; - this.checkpointTimestamp = snapshotTimestamp; - this.latestSequenceNumber = latestSequenceNumber; - this.checkpointCreationTimeTaken = checkpointCreationTimeTaken; -} - -@Override -public Path getCh
[hadoop] branch trunk updated: HDDS-1232. Recon Container DB service definition. Contributed by Aravindan Vijayan.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new fb851c9 HDDS-1232. Recon Container DB service definition. Contributed by Aravindan Vijayan. fb851c9 is described below commit fb851c94817e69ffa75d2b87f496c658c273b73b Author: Yiqun Lin AuthorDate: Fri Mar 8 16:59:41 2019 +0800 HDDS-1232. Recon Container DB service definition. Contributed by Aravindan Vijayan. --- .../apache/hadoop/utils/LevelDBStoreIterator.java | 5 + .../org/apache/hadoop/utils/MetaStoreIterator.java | 5 + .../apache/hadoop/utils/RocksDBStoreIterator.java | 5 + .../common/src/main/resources/ozone-default.xml| 94 + .../org/apache/hadoop/utils/TestMetadataStore.java | 52 .../org/apache/hadoop/hdds/server/ServerUtils.java | 30 +++-- hadoop-ozone/dist/pom.xml | 7 + hadoop-ozone/integration-test/pom.xml | 4 + .../hadoop/ozone/TestOzoneConfigurationFields.java | 2 + hadoop-ozone/ozone-recon/pom.xml | 13 +- ...onControllerModule.java => ReconConstants.java} | 20 ++- .../hadoop/ozone/recon/ReconControllerModule.java | 7 + .../apache/hadoop/ozone/recon/ReconHttpServer.java | 20 +-- ...nfiguration.java => ReconServerConfigKeys.java} | 12 +- .../types/ContainerKeyPrefix.java} | 36 +++-- .../recon/spi/ContainerDBServiceProvider.java | 58 .../ozone/recon/spi/ReconContainerDBProvider.java | 77 +++ .../spi/impl/ContainerDBServiceProviderImpl.java | 138 +++ .../impl/package-info.java}| 20 +-- .../impl/TestContainerDBServiceProviderImpl.java | 148 + .../hadoop/ozone/recon/spi/impl/package-info.java} | 19 +-- hadoop-ozone/pom.xml | 5 + 22 files changed, 697 insertions(+), 80 deletions(-) diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStoreIterator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStoreIterator.java index 7b62f7a..92051dd 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStoreIterator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/LevelDBStoreIterator.java @@ -61,4 +61,9 @@ public class LevelDBStoreIterator implements MetaStoreIterator { public void seekToLast() { levelDBIterator.seekToLast(); } + + @Override + public void prefixSeek(byte[] prefix) { +levelDBIterator.seek(prefix); + } } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetaStoreIterator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetaStoreIterator.java index 52d0a3e..15ded0d 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetaStoreIterator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/MetaStoreIterator.java @@ -36,4 +36,9 @@ public interface MetaStoreIterator extends Iterator { */ void seekToLast(); + /** + * seek with prefix. + */ + void prefixSeek(byte[] prefix); + } diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreIterator.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreIterator.java index 6e9b695..161d5de 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreIterator.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/RocksDBStoreIterator.java @@ -63,4 +63,9 @@ public class RocksDBStoreIterator implements MetaStoreIterator { rocksDBIterator.seekToLast(); } + @Override + public void prefixSeek(byte[] prefix) { +rocksDBIterator.seek(prefix); + } + } diff --git a/hadoop-hdds/common/src/main/resources/ozone-default.xml b/hadoop-hdds/common/src/main/resources/ozone-default.xml index a95d9d1..a0b4c52 100644 --- a/hadoop-hdds/common/src/main/resources/ozone-default.xml +++ b/hadoop-hdds/common/src/main/resources/ozone-default.xml @@ -2144,4 +2144,98 @@ milliseconds. + +ozone.recon.http.enabled +true +RECON, MANAGEMENT + + Property to enable or disable Recon web user interface. + + + +ozone.recon.http-address +0.0.0.0:9888 +RECON, MANAGEMENT + + The address and the base port where the Recon web UI will listen on. + + If the port is 0, then the server will start on a free port. However, it + is best to specify a well-known port, so it is easy to connect and see + the Recon management UI. + + + +ozone.recon.http-bind-host +0.0.0.0 +RECON, MANAGEMENT + + The actual address the Recon server will bind to. If this optional + the address is set, it overrides only the hostname portion of + ozone.recon.http-a
[hadoop] branch trunk updated: HDFS-14182. Datanode usage histogram is clicked to show ip list. Contributed by fengchuang.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 18ea0c1 HDFS-14182. Datanode usage histogram is clicked to show ip list. Contributed by fengchuang. 18ea0c1 is described below commit 18ea0c14933c3e33617647eae2e3076cda1232c0 Author: Yiqun Lin AuthorDate: Mon Mar 4 17:34:24 2019 +0800 HDFS-14182. Datanode usage histogram is clicked to show ip list. Contributed by fengchuang. --- .../hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js | 47 +- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js index 5b2838c..4e8b362 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js @@ -303,10 +303,13 @@ .attr("class", "bar") .attr("transform", function(d) { return "translate(" + x(d.x0) + "," + y(d.length) + ")"; }); + window.liveNodes = dnData.LiveNodes; + bar.append("rect") .attr("x", 1) .attr("width", x(bins[0].x1) - x(bins[0].x0) - 1) - .attr("height", function(d) { return height - y(d.length); }); + .attr("height", function(d) { return height - y(d.length); }) + .attr("onclick", function (d) { return "open_hostip_list(" + d.x0 + "," + d.x1 + ")"; }); bar.append("text") .attr("dy", ".75em") @@ -425,3 +428,45 @@ load_page(); }); })(); + +function open_hostip_list(x0, x1) { + close_hostip_list(); + var ips = new Array(); + for (var i = 0; i < liveNodes.length; i++) { +var dn = liveNodes[i]; +var index = (dn.usedSpace / dn.capacity) * 100.0; +if (index == 0) { + index = 1; +} +//More than 100% do not care,so not record in 95%-100% bar +if (index > x0 && index <= x1) { + ips.push(dn.infoAddr.split(":")[0]); +} + } + var ipsText = ''; + for (var i = 0; i < ips.length; i++) { +ipsText += ips[i] + '\n'; + } + var histogram_div = document.getElementById('datanode-usage-histogram'); + histogram_div.setAttribute('style', 'position: relative'); + var ips_div = document.createElement("textarea"); + ips_div.setAttribute('id', 'datanode_ips'); + ips_div.setAttribute('rows', '8'); + ips_div.setAttribute('cols', '14'); + ips_div.setAttribute('style', 'position: absolute;top: 0px;right: -38px;'); + ips_div.setAttribute('readonly', 'readonly'); + histogram_div.appendChild(ips_div); + + var close_div = document.createElement("div"); + histogram_div.appendChild(close_div); + close_div.setAttribute('id', 'close_ips'); + close_div.setAttribute('style', 'position: absolute;top: 0px;right: -62px;width:20px;height;20px'); + close_div.setAttribute('onclick', 'close_hostip_list()'); + close_div.innerHTML = "X"; + ips_div.innerHTML = ipsText; +} + +function close_hostip_list() { + $("#datanode_ips").remove(); + $("#close_ips").remove(); +} \ No newline at end of file - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDDS-1130. Make BenchMarkBlockManager multi-threaded. Contributed by Lokesh Jain.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 1d30fd9 HDDS-1130. Make BenchMarkBlockManager multi-threaded. Contributed by Lokesh Jain. 1d30fd9 is described below commit 1d30fd94c6430492ce2f92883117eff56094eec0 Author: Yiqun Lin AuthorDate: Wed Feb 20 10:45:51 2019 +0800 HDDS-1130. Make BenchMarkBlockManager multi-threaded. Contributed by Lokesh Jain. --- .../ozone/genesis/BenchMarkBlockManager.java | 81 +- 1 file changed, 49 insertions(+), 32 deletions(-) diff --git a/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkBlockManager.java b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkBlockManager.java index 4f7e096..cc08709 100644 --- a/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkBlockManager.java +++ b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkBlockManager.java @@ -19,6 +19,7 @@ package org.apache.hadoop.ozone.genesis; import org.apache.commons.lang3.RandomStringUtils; +import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.hdds.HddsConfigKeys; import org.apache.hadoop.hdds.conf.OzoneConfiguration; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor; @@ -46,6 +47,7 @@ import java.io.IOException; import java.util.UUID; import java.util.List; import java.util.ArrayList; +import java.util.concurrent.locks.ReentrantLock; import static org.apache.hadoop.hdds.scm.ScmConfigKeys.OZONE_SCM_DB_CACHE_SIZE_DEFAULT; import static org.apache.hadoop.hdds.scm.ScmConfigKeys.OZONE_SCM_DB_CACHE_SIZE_MB; @@ -59,9 +61,16 @@ import static org.apache.hadoop.ozone.OzoneConsts.SCM_PIPELINE_DB; @State(Scope.Thread) public class BenchMarkBlockManager { - private StorageContainerManager scm; - private PipelineManager pipelineManager; - private BlockManager blockManager; + private static String testDir; + private static StorageContainerManager scm; + private static PipelineManager pipelineManager; + private static BlockManager blockManager; + private static ReentrantLock lock = new ReentrantLock(); + + @Param({"1", "10", "100", "1000", "1", "10"}) + private static int numPipelines; + @Param({"3", "10", "100"}) + private static int numContainersPerPipeline; private static StorageContainerManager getScm(OzoneConfiguration conf, SCMConfigurator configurator) throws IOException, @@ -80,46 +89,53 @@ public class BenchMarkBlockManager { } @Setup(Level.Trial) - public void initialize() + public static void initialize() throws IOException, AuthenticationException, InterruptedException { -OzoneConfiguration conf = new OzoneConfiguration(); -conf.set(HddsConfigKeys.OZONE_METADATA_DIRS, -GenesisUtil.getTempPath().resolve(RandomStringUtils.randomNumeric(7)) -.toString()); -conf.setInt(OZONE_SCM_PIPELINE_OWNER_CONTAINER_COUNT, 100); -final File metaDir = ServerUtils.getScmDbDir(conf); -final File pipelineDBPath = new File(metaDir, SCM_PIPELINE_DB); -int cacheSize = conf.getInt(OZONE_SCM_DB_CACHE_SIZE_MB, -OZONE_SCM_DB_CACHE_SIZE_DEFAULT); -MetadataStore pipelineStore = -MetadataStoreBuilder.newBuilder() -.setCreateIfMissing(true) -.setConf(conf) -.setDbFile(pipelineDBPath) -.setCacheSize(cacheSize * OzoneConsts.MB) -.build(); -addPipelines(100, ReplicationFactor.THREE, pipelineStore); -pipelineStore.close(); -scm = getScm(conf, new SCMConfigurator()); -pipelineManager = scm.getPipelineManager(); -for (Pipeline pipeline : pipelineManager -.getPipelines(ReplicationType.RATIS, ReplicationFactor.THREE)) { - pipelineManager.openPipeline(pipeline.getId()); +try { + lock.lock(); + if (scm == null) { +OzoneConfiguration conf = new OzoneConfiguration(); +testDir = GenesisUtil.getTempPath() +.resolve(RandomStringUtils.randomNumeric(7)).toString(); +conf.set(HddsConfigKeys.OZONE_METADATA_DIRS, testDir); +conf.setInt(OZONE_SCM_PIPELINE_OWNER_CONTAINER_COUNT, +numContainersPerPipeline); +final File metaDir = ServerUtils.getScmDbDir(conf); +final File pipelineDBPath = new File(metaDir, SCM_PIPELINE_DB); +int cacheSize = conf.getInt(OZONE_SCM_DB_CACHE_SIZE_MB, +OZONE_SCM_DB_CACHE_SIZE_DEFAULT); +MetadataStore pipelineStore = +MetadataStoreBuilder.newBuilder().setCreateIfMissing(true) +.setConf(conf).setDbFile(pipelineDBPath) +.setCacheSize(cacheSize * OzoneConsts.MB).build(); +addPipelines(Replicat
[hadoop] branch trunk updated: HDDS-1122. Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure. Contributed by Yiqun Lin.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 67af509 HDDS-1122. Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure. Contributed by Yiqun Lin. 67af509 is described below commit 67af509097d8df4df161d53e3e511b312e8ac3dd Author: Yiqun Lin AuthorDate: Tue Feb 19 11:29:52 2019 +0800 HDDS-1122. Fix TestOzoneManagerRatisServer#testSubmitRatisRequest unit test failure. Contributed by Yiqun Lin. --- .../src/main/java/org/apache/hadoop/ozone/om/ratis/OMRatisHelper.java | 2 ++ 1 file changed, 2 insertions(+) diff --git a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OMRatisHelper.java b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OMRatisHelper.java index ee1fee6..9115421 100644 --- a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OMRatisHelper.java +++ b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OMRatisHelper.java @@ -25,6 +25,7 @@ import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos .OMRequest; import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos .OMResponse; +import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.Status; import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.Type; import org.apache.ratis.RaftConfigKeys; import org.apache.ratis.client.RaftClient; @@ -110,6 +111,7 @@ public final class OMRatisHelper { .setCmdType(cmdType) .setSuccess(false) .setMessage(e.getMessage()) +.setStatus(Status.INTERNAL_ERROR) .build(); } } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDDS-1106. Introduce queryMap in PipelineManager. Contributed by Lokesh Jain.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new f2fb653 HDDS-1106. Introduce queryMap in PipelineManager. Contributed by Lokesh Jain. f2fb653 is described below commit f2fb6536dcbe6320f69273bf9e11d4701248172c Author: Yiqun Lin AuthorDate: Mon Feb 18 22:35:23 2019 +0800 HDDS-1106. Introduce queryMap in PipelineManager. Contributed by Lokesh Jain. --- .../hadoop/hdds/scm/pipeline/PipelineStateMap.java | 72 +- .../scm/pipeline/TestPipelineStateManager.java | 42 + 2 files changed, 111 insertions(+), 3 deletions(-) diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateMap.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateMap.java index dea2115..2b6c61b 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateMap.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateMap.java @@ -18,6 +18,7 @@ package org.apache.hadoop.hdds.scm.pipeline; import com.google.common.base.Preconditions; +import org.apache.commons.lang3.builder.HashCodeBuilder; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationType; import org.apache.hadoop.hdds.scm.container.ContainerID; @@ -27,6 +28,7 @@ import org.slf4j.LoggerFactory; import java.io.IOException; import java.util.*; +import java.util.concurrent.CopyOnWriteArrayList; import java.util.stream.Collectors; /** @@ -42,15 +44,27 @@ class PipelineStateMap { private final Map pipelineMap; private final Map> pipeline2container; + private final Map> query2OpenPipelines; PipelineStateMap() { // TODO: Use TreeMap for range operations? -this.pipelineMap = new HashMap<>(); -this.pipeline2container = new HashMap<>(); +pipelineMap = new HashMap<>(); +pipeline2container = new HashMap<>(); +query2OpenPipelines = new HashMap<>(); +initializeQueryMap(); } + private void initializeQueryMap() { +for (ReplicationType type : ReplicationType.values()) { + for (ReplicationFactor factor : ReplicationFactor.values()) { +query2OpenPipelines +.put(new PipelineQuery(type, factor), new CopyOnWriteArrayList<>()); + } +} + } + /** * Adds provided pipeline in the data structures. * @@ -70,6 +84,9 @@ class PipelineStateMap { .format("Duplicate pipeline ID %s detected.", pipeline.getId())); } pipeline2container.put(pipeline.getId(), new TreeSet<>()); +if (pipeline.getPipelineState() == PipelineState.OPEN) { + query2OpenPipelines.get(new PipelineQuery(pipeline)).add(pipeline); +} } /** @@ -188,6 +205,10 @@ class PipelineStateMap { Preconditions.checkNotNull(factor, "Replication factor cannot be null"); Preconditions.checkNotNull(state, "Pipeline state cannot be null"); +if (state == PipelineState.OPEN) { + return Collections.unmodifiableList( + query2OpenPipelines.get(new PipelineQuery(type, factor))); +} return pipelineMap.values().stream().filter( pipeline -> pipeline.getType() == type && pipeline.getPipelineState() == state @@ -293,7 +314,52 @@ class PipelineStateMap { Preconditions.checkNotNull(state, "Pipeline LifeCycleState cannot be null"); final Pipeline pipeline = getPipeline(pipelineID); -return pipelineMap.compute(pipelineID, +Pipeline updatedPipeline = pipelineMap.compute(pipelineID, (id, p) -> Pipeline.newBuilder(pipeline).setState(state).build()); +PipelineQuery query = new PipelineQuery(pipeline); +if (updatedPipeline.getPipelineState() == PipelineState.OPEN) { + // for transition to OPEN state add pipeline to query2OpenPipelines + query2OpenPipelines.get(query).add(updatedPipeline); +} else if (updatedPipeline.getPipelineState() == PipelineState.CLOSED) { + // for transition from OPEN to CLOSED state remove pipeline from + // query2OpenPipelines + query2OpenPipelines.get(query).remove(pipeline); +} +return updatedPipeline; + } + + private class PipelineQuery { +private ReplicationType type; +private ReplicationFactor factor; + +PipelineQuery(ReplicationType type, ReplicationFactor factor) { + this.type = type; + this.factor = factor; +} + +PipelineQuery(Pipeline pipeline) { + type = pipeline.getType(); + factor = pipeline.getFactor(); +} + +@Override +public boolean equals(Object other) { + if (this == other) { +
[hadoop] 02/02: HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git commit 084b6a6751dd203de1c7f3c65077ca72f1d83632 Author: Yiqun Lin AuthorDate: Fri Feb 15 14:23:34 2019 +0800 HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham. --- hadoop-ozone/tools/pom.xml | 9 +- .../ozone/genesis/BenchMarkOMKeyAllocation.java| 135 + .../org/apache/hadoop/ozone/genesis/Genesis.java | 1 + 3 files changed, 140 insertions(+), 5 deletions(-) diff --git a/hadoop-ozone/tools/pom.xml b/hadoop-ozone/tools/pom.xml index 95bef70..aeff0f7 100644 --- a/hadoop-ozone/tools/pom.xml +++ b/hadoop-ozone/tools/pom.xml @@ -31,6 +31,10 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd";> org.apache.hadoop + hadoop-ozone-ozone-manager + + + org.apache.hadoop hadoop-ozone-common @@ -78,11 +82,6 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd";> 2.15.0 test - - org.apache.hadoop - hadoop-ozone-ozone-manager - 0.4.0-SNAPSHOT - diff --git a/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkOMKeyAllocation.java b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkOMKeyAllocation.java new file mode 100644 index 000..fbb686a --- /dev/null +++ b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkOMKeyAllocation.java @@ -0,0 +1,135 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.genesis; + +import org.apache.commons.io.FileUtils; +import org.apache.commons.lang3.RandomStringUtils; +import org.apache.commons.lang3.RandomUtils; +import org.apache.hadoop.hdds.client.BlockID; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; +import org.apache.hadoop.ozone.om.BucketManager; +import org.apache.hadoop.ozone.om.BucketManagerImpl; +import org.apache.hadoop.ozone.om.KeyManager; +import org.apache.hadoop.ozone.om.KeyManagerImpl; +import org.apache.hadoop.ozone.om.OMConfigKeys; +import org.apache.hadoop.ozone.om.OmMetadataManagerImpl; +import org.apache.hadoop.ozone.om.VolumeManager; +import org.apache.hadoop.ozone.om.VolumeManagerImpl; +import org.apache.hadoop.ozone.om.helpers.OmBucketInfo; +import org.apache.hadoop.ozone.om.helpers.OmKeyArgs; +import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo; +import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs; +import org.apache.hadoop.ozone.om.helpers.OpenKeySession; +import org.apache.hadoop.security.UserGroupInformation; +import org.openjdk.jmh.annotations.Benchmark; +import org.openjdk.jmh.annotations.Level; +import org.openjdk.jmh.annotations.Scope; +import org.openjdk.jmh.annotations.Setup; +import org.openjdk.jmh.annotations.State; +import org.openjdk.jmh.annotations.TearDown; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; + +/** + * Benchmark key creation in a bucket in OM. + */ +@State(Scope.Thread) +public class BenchMarkOMKeyAllocation { + + private static final String TMP_DIR = "java.io.tmpdir"; + private String volumeName = UUID.randomUUID().toString(); + private String bucketName = UUID.randomUUID().toString(); + private KeyManager keyManager; + private VolumeManager volumeManager; + private BucketManager bucketManager; + private String path = Paths.get(System.getProperty(TMP_DIR)).resolve( + RandomStringUtils.randomNumeric(6)).toFile() +.getAbsolutePath(); + + @Setup(Level.Trial) + public void setup() throws IOException { +OzoneConfiguration configuration = new OzoneConfiguration(); +configuration.set(OMConfigKeys.OZONE_OM_DB_DIRS, path); + +OmMetadataManagerImpl omMetadataManager = +new OmMetadataManagerImpl(configuration); +volumeManager = new VolumeManagerImpl(omMetadataManager, configuration); +bucketMan
[hadoop] branch trunk updated (5656409 -> 084b6a6)
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 5656409 HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham. new 492e49e Revert "HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham." new 084b6a6 HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../tools/src/main/java/org/apache/hadoop/ozone/genesis/Genesis.java | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] 01/02: Revert "HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham."
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git commit 492e49e7caff34231b07c85a0038f27f41de67f7 Author: Yiqun Lin AuthorDate: Fri Feb 15 14:21:29 2019 +0800 Revert "HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham." This reverts commit 5656409327db5a590cc29b846d291dad005bf8d0. --- hadoop-ozone/tools/pom.xml | 9 +- .../ozone/genesis/BenchMarkOMKeyAllocation.java| 135 - 2 files changed, 5 insertions(+), 139 deletions(-) diff --git a/hadoop-ozone/tools/pom.xml b/hadoop-ozone/tools/pom.xml index aeff0f7..95bef70 100644 --- a/hadoop-ozone/tools/pom.xml +++ b/hadoop-ozone/tools/pom.xml @@ -31,10 +31,6 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd";> org.apache.hadoop - hadoop-ozone-ozone-manager - - - org.apache.hadoop hadoop-ozone-common @@ -82,6 +78,11 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd";> 2.15.0 test + + org.apache.hadoop + hadoop-ozone-ozone-manager + 0.4.0-SNAPSHOT + diff --git a/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkOMKeyAllocation.java b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkOMKeyAllocation.java deleted file mode 100644 index fbb686a..000 --- a/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkOMKeyAllocation.java +++ /dev/null @@ -1,135 +0,0 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.hadoop.ozone.genesis; - -import org.apache.commons.io.FileUtils; -import org.apache.commons.lang3.RandomStringUtils; -import org.apache.commons.lang3.RandomUtils; -import org.apache.hadoop.hdds.client.BlockID; -import org.apache.hadoop.hdds.conf.OzoneConfiguration; -import org.apache.hadoop.hdds.protocol.proto.HddsProtos; -import org.apache.hadoop.ozone.om.BucketManager; -import org.apache.hadoop.ozone.om.BucketManagerImpl; -import org.apache.hadoop.ozone.om.KeyManager; -import org.apache.hadoop.ozone.om.KeyManagerImpl; -import org.apache.hadoop.ozone.om.OMConfigKeys; -import org.apache.hadoop.ozone.om.OmMetadataManagerImpl; -import org.apache.hadoop.ozone.om.VolumeManager; -import org.apache.hadoop.ozone.om.VolumeManagerImpl; -import org.apache.hadoop.ozone.om.helpers.OmBucketInfo; -import org.apache.hadoop.ozone.om.helpers.OmKeyArgs; -import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo; -import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs; -import org.apache.hadoop.ozone.om.helpers.OpenKeySession; -import org.apache.hadoop.security.UserGroupInformation; -import org.openjdk.jmh.annotations.Benchmark; -import org.openjdk.jmh.annotations.Level; -import org.openjdk.jmh.annotations.Scope; -import org.openjdk.jmh.annotations.Setup; -import org.openjdk.jmh.annotations.State; -import org.openjdk.jmh.annotations.TearDown; - -import java.io.File; -import java.io.IOException; -import java.nio.file.Paths; -import java.util.ArrayList; -import java.util.List; -import java.util.UUID; - -/** - * Benchmark key creation in a bucket in OM. - */ -@State(Scope.Thread) -public class BenchMarkOMKeyAllocation { - - private static final String TMP_DIR = "java.io.tmpdir"; - private String volumeName = UUID.randomUUID().toString(); - private String bucketName = UUID.randomUUID().toString(); - private KeyManager keyManager; - private VolumeManager volumeManager; - private BucketManager bucketManager; - private String path = Paths.get(System.getProperty(TMP_DIR)).resolve( - RandomStringUtils.randomNumeric(6)).toFile() -.getAbsolutePath(); - - @Setup(Level.Trial) - public void setup() throws IOException { -OzoneConfiguration configuration = new OzoneConfiguration(); -configuration.set(OMConfigKeys.OZONE_OM_DB_DIRS, path); - -OmMetadataManagerImpl omMetadataManager = -new OmMetadataManagerImpl(configuration); -volumeManager = new VolumeManagerImpl(omMetadataManager, co
[hadoop] branch trunk updated: HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 5656409 HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham. 5656409 is described below commit 5656409327db5a590cc29b846d291dad005bf8d0 Author: Yiqun Lin AuthorDate: Fri Feb 15 14:07:15 2019 +0800 HDDS-1099. Genesis benchmark for ozone key creation in OM. Contributed by Bharat Viswanadham. --- hadoop-ozone/tools/pom.xml | 9 +- .../ozone/genesis/BenchMarkOMKeyAllocation.java| 135 + 2 files changed, 139 insertions(+), 5 deletions(-) diff --git a/hadoop-ozone/tools/pom.xml b/hadoop-ozone/tools/pom.xml index 95bef70..aeff0f7 100644 --- a/hadoop-ozone/tools/pom.xml +++ b/hadoop-ozone/tools/pom.xml @@ -31,6 +31,10 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd";> org.apache.hadoop + hadoop-ozone-ozone-manager + + + org.apache.hadoop hadoop-ozone-common @@ -78,11 +82,6 @@ http://maven.apache.org/xsd/maven-4.0.0.xsd";> 2.15.0 test - - org.apache.hadoop - hadoop-ozone-ozone-manager - 0.4.0-SNAPSHOT - diff --git a/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkOMKeyAllocation.java b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkOMKeyAllocation.java new file mode 100644 index 000..fbb686a --- /dev/null +++ b/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkOMKeyAllocation.java @@ -0,0 +1,135 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.ozone.genesis; + +import org.apache.commons.io.FileUtils; +import org.apache.commons.lang3.RandomStringUtils; +import org.apache.commons.lang3.RandomUtils; +import org.apache.hadoop.hdds.client.BlockID; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; +import org.apache.hadoop.ozone.om.BucketManager; +import org.apache.hadoop.ozone.om.BucketManagerImpl; +import org.apache.hadoop.ozone.om.KeyManager; +import org.apache.hadoop.ozone.om.KeyManagerImpl; +import org.apache.hadoop.ozone.om.OMConfigKeys; +import org.apache.hadoop.ozone.om.OmMetadataManagerImpl; +import org.apache.hadoop.ozone.om.VolumeManager; +import org.apache.hadoop.ozone.om.VolumeManagerImpl; +import org.apache.hadoop.ozone.om.helpers.OmBucketInfo; +import org.apache.hadoop.ozone.om.helpers.OmKeyArgs; +import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo; +import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs; +import org.apache.hadoop.ozone.om.helpers.OpenKeySession; +import org.apache.hadoop.security.UserGroupInformation; +import org.openjdk.jmh.annotations.Benchmark; +import org.openjdk.jmh.annotations.Level; +import org.openjdk.jmh.annotations.Scope; +import org.openjdk.jmh.annotations.Setup; +import org.openjdk.jmh.annotations.State; +import org.openjdk.jmh.annotations.TearDown; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; + +/** + * Benchmark key creation in a bucket in OM. + */ +@State(Scope.Thread) +public class BenchMarkOMKeyAllocation { + + private static final String TMP_DIR = "java.io.tmpdir"; + private String volumeName = UUID.randomUUID().toString(); + private String bucketName = UUID.randomUUID().toString(); + private KeyManager keyManager; + private VolumeManager volumeManager; + private BucketManager bucketManager; + private String path = Paths.get(System.getProperty(TMP_DIR)).resolve( + RandomStringUtils.randomNumeric(6)).toFile() +.getAbsolutePath(); + + @Setup(Level.Trial) + public void setup() throws IOException { +OzoneConfiguration configuration = new OzoneConfiguration(); +configuration.set(OMConfigKeys.OZONE_OM_DB_DIRS, path); + +OmMetadataManagerImpl omMetadataManager = +
[hadoop] branch trunk updated: HADOOP-16097. Provide proper documentation for FairCallQueue. Contributed by Erik Krogen.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 7b11b40 HADOOP-16097. Provide proper documentation for FairCallQueue. Contributed by Erik Krogen. 7b11b40 is described below commit 7b11b404a35f93e4b4b12546034ef8001720eb5f Author: Yiqun Lin AuthorDate: Wed Feb 13 11:16:04 2019 +0800 HADOOP-16097. Provide proper documentation for FairCallQueue. Contributed by Erik Krogen. --- .../src/site/markdown/FairCallQueue.md | 150 + .../resources/images/faircallqueue-overview.png| Bin 0 -> 47397 bytes hadoop-project/src/site/site.xml | 1 + 3 files changed, 151 insertions(+) diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/FairCallQueue.md b/hadoop-common-project/hadoop-common/src/site/markdown/FairCallQueue.md new file mode 100644 index 000..e62c7ad --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/FairCallQueue.md @@ -0,0 +1,150 @@ + + +Fair Call Queue Guide += + + + +Purpose +--- + +This document describes how to configure and manage the Fair Call Queue for Hadoop. + +Prerequisites +- + +Make sure Hadoop is installed, configured and setup correctly. For more information see: + +* [Single Node Setup](./SingleCluster.html) for first-time users. +* [Cluster Setup](./ClusterSetup.html) for large, distributed clusters. + +Overview + + +Hadoop server components, in particular the HDFS NameNode, experience very heavy RPC load from clients. By default, +all client requests are routed through a first-in, first-out queue and serviced in the order they arrive. This means +that a single user submitting a very large number of requests can easily overwhelm the service, causing degraded service +for all other users. The Fair Call Queue, and related components, aim to mitigate this impact. + +Design Details +-- + +There are a few components in the IPC stack which have a complex interplay, each with their own tuning parameters. +The image below presents a schematic overview of their interactions, which will be explained below. + +![FairCallQueue Overview](./images/faircallqueue-overview.png) + +In the following explanation, **bolded** words refer to named entities or configurables. + +When a client makes a request to an IPC server, this request first lands in a **listen queue**. **Reader** threads +remove requests from this queue and pass them to a configurable **RpcScheduler** to be assigned a priority and placed +into a **call queue**; this is where FairCallQueue sits as a pluggable implementation (the other existing +implementation being a FIFO queue). **Handler** threads accept requests out of the call queue, process them, and +respond to the client. + +The implementation of RpcScheduler used with FairCallQueue by default is **DecayRpcScheduler**, which maintains a +count of requests received for each user. This count _decays_ over time; every **sweep period** (5s by default), +the number of requests per user is multiplied by a **decay factor** (0.5 by default). This maintains a weighted/rolling +average of request count per user. Every time that a sweep is performed, the call counts for all known users are +ranked from highest to lowest. Each user is assigned a **priority** (0-3 by default, with 0 being highest priority) +based on the proportion of calls originating from that user. The default **priority thresholds** are (0.125, 0.25, 0.5), +meaning that users whose calls make up more than 50% of the total (there can be at most one such user) are placed into +the lowest priority, users whose calls make up between 25% and 50% of the total are in the 2nd lowest, users whose calls +make up between 12.5% and 25% are in the 2nd highest priority, and all other users are placed in the highest priority. +At the end of the sweep, each known user has a cached priority which will be used until the next sweep; new users which +appear between sweeps will have their priority calculated on-the-fly. + +Within FairCallQueue, there are multiple **priority queues**, each of which is designated a **weight**. When a request +arrives at the call queue, the request is placed into one of these priority queues based on the current priority +assigned to the call (by the RpcScheduler). When a handler thread attempts to fetch an item from the call queue, which +queue it pulls from is decided via an **RpcMultiplexer**; currently this is hard-coded to be a +**WeightedRoundRobinMultiplexer**. The WRRM serves requests from queues based on their weights; the default weights +for the default 4 priority levels are (8, 4, 2, 1). Thus, the WRRM would serve 8 requests from the highest priority +queue, 4 from the second highest, 2 from the third highest
[hadoop] branch trunk updated: HDDS-1047. Fix TestRatisPipelineProvider#testCreatePipelineWithFactor. Contributed by Nilotpal Nandi.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 06d7890 HDDS-1047. Fix TestRatisPipelineProvider#testCreatePipelineWithFactor. Contributed by Nilotpal Nandi. 06d7890 is described below commit 06d7890bdd3e597824f9ca02b453d45eef445f49 Author: Yiqun Lin AuthorDate: Wed Feb 13 10:50:57 2019 +0800 HDDS-1047. Fix TestRatisPipelineProvider#testCreatePipelineWithFactor. Contributed by Nilotpal Nandi. --- .../scm/pipeline/TestRatisPipelineProvider.java| 32 ++ 1 file changed, 32 insertions(+) diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineProvider.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineProvider.java index 6f4934f..6f385de 100644 --- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineProvider.java +++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineProvider.java @@ -50,6 +50,28 @@ public class TestRatisPipelineProvider { stateManager, new OzoneConfiguration()); } + private void createPipelineAndAssertions( + HddsProtos.ReplicationFactor factor) throws IOException { +Pipeline pipeline = provider.create(factor); +stateManager.addPipeline(pipeline); +Assert.assertEquals(pipeline.getType(), HddsProtos.ReplicationType.RATIS); +Assert.assertEquals(pipeline.getFactor(), factor); +Assert.assertEquals(pipeline.getPipelineState(), +Pipeline.PipelineState.OPEN); +Assert.assertEquals(pipeline.getNodes().size(), factor.getNumber()); +Pipeline pipeline1 = provider.create(factor); +stateManager.addPipeline(pipeline1); +// New pipeline should not overlap with the previous created pipeline +Assert.assertTrue( +CollectionUtils.intersection(pipeline.getNodes(), pipeline1.getNodes()) +.isEmpty()); +Assert.assertEquals(pipeline1.getType(), HddsProtos.ReplicationType.RATIS); +Assert.assertEquals(pipeline1.getFactor(), factor); +Assert.assertEquals(pipeline1.getPipelineState(), +Pipeline.PipelineState.OPEN); +Assert.assertEquals(pipeline1.getNodes().size(), factor.getNumber()); + } + @Test public void testCreatePipelineWithFactor() throws IOException { HddsProtos.ReplicationFactor factor = HddsProtos.ReplicationFactor.THREE; @@ -76,6 +98,16 @@ public class TestRatisPipelineProvider { Assert.assertEquals(pipeline1.getNodes().size(), factor.getNumber()); } + @Test + public void testCreatePipelineWithFactorThree() throws IOException { +createPipelineAndAssertions(HddsProtos.ReplicationFactor.THREE); + } + + @Test + public void testCreatePipelineWithFactorOne() throws IOException { +createPipelineAndAssertions(HddsProtos.ReplicationFactor.ONE); + } + private List createListOfNodes(int nodeCount) { List nodes = new ArrayList<>(); for (int i = 0; i < nodeCount; i++) { - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDDS-1074. Remove dead variable from KeyOutputStream#addKeyLocationInfo. Contributed by Siddharth Wagle.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 26e6013 HDDS-1074. Remove dead variable from KeyOutputStream#addKeyLocationInfo. Contributed by Siddharth Wagle. 26e6013 is described below commit 26e60135f57868cf7d1e3a0229c55ed045f9745b Author: Yiqun Lin AuthorDate: Tue Feb 12 16:28:50 2019 +0800 HDDS-1074. Remove dead variable from KeyOutputStream#addKeyLocationInfo. Contributed by Siddharth Wagle. --- .../main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java | 3 --- 1 file changed, 3 deletions(-) diff --git a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java index af39631..b94e14f 100644 --- a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java +++ b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java @@ -21,7 +21,6 @@ import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; import org.apache.hadoop.fs.FSExceptionMessages; import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.Result; -import org.apache.hadoop.hdds.scm.XceiverClientSpi; import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException; import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerWithPipeline; import org.apache.hadoop.ozone.common.Checksum; @@ -202,8 +201,6 @@ public class KeyOutputStream extends OutputStream { ContainerWithPipeline containerWithPipeline = scmClient .getContainerWithPipeline(subKeyInfo.getContainerID()); UserGroupInformation.getCurrentUser().addToken(subKeyInfo.getToken()); -XceiverClientSpi xceiverClient = - xceiverClientManager.acquireClient(containerWithPipeline.getPipeline()); BlockOutputStreamEntry.Builder builder = new BlockOutputStreamEntry.Builder() .setBlockID(subKeyInfo.getBlockID()) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDDS-1048. Remove SCMNodeStat from SCMNodeManager and use storage information from DatanodeInfo#StorageReportProto. Contributed by Nanda kumar.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new fb8c997 HDDS-1048. Remove SCMNodeStat from SCMNodeManager and use storage information from DatanodeInfo#StorageReportProto. Contributed by Nanda kumar. fb8c997 is described below commit fb8c997a6884bbe19c45bab77950068bb78109c7 Author: Yiqun Lin AuthorDate: Fri Feb 8 23:49:37 2019 +0800 HDDS-1048. Remove SCMNodeStat from SCMNodeManager and use storage information from DatanodeInfo#StorageReportProto. Contributed by Nanda kumar. --- .../apache/hadoop/hdds/scm/node/DatanodeInfo.java | 7 +- .../hadoop/hdds/scm/node/DeadNodeHandler.java | 1 - .../apache/hadoop/hdds/scm/node/NodeManager.java | 15 +- .../hadoop/hdds/scm/node/NodeStateManager.java | 65 ++-- .../hadoop/hdds/scm/node/SCMNodeManager.java | 169 ++--- .../hadoop/hdds/scm/node/states/NodeStateMap.java | 58 --- .../hadoop/hdds/scm/container/MockNodeManager.java | 31 +--- .../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 28 ++-- .../hdds/scm/node/TestNodeReportHandler.java | 3 +- .../testutils/ReplicationNodeManagerMock.java | 11 +- 10 files changed, 123 insertions(+), 265 deletions(-) diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DatanodeInfo.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DatanodeInfo.java index 26b8b95..d06ea2a 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DatanodeInfo.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DatanodeInfo.java @@ -23,6 +23,7 @@ import org.apache.hadoop.hdds.protocol.proto .StorageContainerDatanodeProtocolProtos.StorageReportProto; import org.apache.hadoop.util.Time; +import java.util.Collections; import java.util.List; import java.util.concurrent.locks.ReadWriteLock; import java.util.concurrent.locks.ReentrantReadWriteLock; @@ -38,7 +39,6 @@ public class DatanodeInfo extends DatanodeDetails { private volatile long lastHeartbeatTime; private long lastStatsUpdatedTime; - // If required we can dissect StorageReportProto and store the raw data private List storageReports; /** @@ -48,8 +48,9 @@ public class DatanodeInfo extends DatanodeDetails { */ public DatanodeInfo(DatanodeDetails datanodeDetails) { super(datanodeDetails); -lock = new ReentrantReadWriteLock(); -lastHeartbeatTime = Time.monotonicNow(); +this.lock = new ReentrantReadWriteLock(); +this.lastHeartbeatTime = Time.monotonicNow(); +this.storageReports = Collections.emptyList(); } /** diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DeadNodeHandler.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DeadNodeHandler.java index 8e71399..a75a51a 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DeadNodeHandler.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DeadNodeHandler.java @@ -58,7 +58,6 @@ public class DeadNodeHandler implements EventHandler { @Override public void onMessage(DatanodeDetails datanodeDetails, EventPublisher publisher) { -nodeManager.processDeadNode(datanodeDetails.getUuid()); // TODO: check if there are any pipeline on this node and fire close // pipeline event diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java index d8865a8..6b8d477 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java @@ -93,8 +93,7 @@ public interface NodeManager extends StorageContainerNodeProtocol, * Return a map of node stats. * @return a map of individual node stats (live/stale but not dead). */ - // TODO: try to change the return type to Map - Map getNodeStats(); + Map getNodeStats(); /** * Return the node stat of the specified datanode. @@ -159,17 +158,11 @@ public interface NodeManager extends StorageContainerNodeProtocol, /** * Process node report. * - * @param dnUuid + * @param datanodeDetails * @param nodeReport */ - void processNodeReport(DatanodeDetails dnUuid, NodeReportProto nodeReport); - - /** - * Process a dead node event in this Node Manager. - * - * @param dnUuid datanode uuid. - */ - void processDeadNode(UUID dnUuid); + void processNodeReport(DatanodeDetails datanodeDetails, + NodeReportProto nodeReport); /** * Get list of SCMCommands in the Command Queue for a particular Datanode. diff --git a
[hadoop] branch trunk updated: HDFS-14172. Avoid NPE when SectionName#fromString returns null. Contributed by Xiang Li.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 1771317 HDFS-14172. Avoid NPE when SectionName#fromString returns null. Contributed by Xiang Li. 1771317 is described below commit 177131793a88960b734038f6e646476d568c3626 Author: Yiqun Lin AuthorDate: Fri Feb 8 20:51:30 2019 +0800 HDFS-14172. Avoid NPE when SectionName#fromString returns null. Contributed by Xiang Li. --- .../apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java | 7 +-- .../hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java | 8 +++- .../hadoop/hdfs/tools/offlineImageViewer/PBImageTextWriter.java | 6 +- .../hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java| 6 +- 4 files changed, 22 insertions(+), 5 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java index 7aed5fd..ad883b1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java @@ -241,8 +241,11 @@ public final class FSImageFormatProtobuf { summary.getCodec(), in); String n = s.getName(); - -switch (SectionName.fromString(n)) { +SectionName sectionName = SectionName.fromString(n); +if (sectionName == null) { + throw new IOException("Unrecognized section " + n); +} +switch (sectionName) { case NS_INFO: loadNameSystemSection(in); break; diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java index 7152c88..f65d29c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FSImageLoader.java @@ -156,7 +156,13 @@ class FSImageLoader { LOG.debug("Loading section " + s.getName() + " length: " + s.getLength ()); } -switch (FSImageFormatProtobuf.SectionName.fromString(s.getName())) { + +FSImageFormatProtobuf.SectionName sectionName += FSImageFormatProtobuf.SectionName.fromString(s.getName()); +if (sectionName == null) { + throw new IOException("Unrecognized section " + s.getName()); +} +switch (sectionName) { case STRING_TABLE: stringTable = loadStringTable(is); break; diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageTextWriter.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageTextWriter.java index 6f36be4..c5bf5af 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageTextWriter.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageTextWriter.java @@ -594,7 +594,11 @@ abstract class PBImageTextWriter implements Closeable { is = FSImageUtil.wrapInputStreamForCompression(conf, summary.getCodec(), new BufferedInputStream(new LimitInputStream( fin, section.getLength(; -switch (SectionName.fromString(section.getName())) { +SectionName sectionName = SectionName.fromString(section.getName()); +if (sectionName == null) { + throw new IOException("Unrecognized section " + section.getName()); +} +switch (sectionName) { case STRING_TABLE: LOG.info("Loading string table"); stringTable = FSImageLoader.loadStringTable(is); diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java index 7a5ef31..cec44f5 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java @@ -326,7 +326,11 @@ public final class PBImageXmlWriter { summary.getCodec(), new BufferedInputSt
[hadoop] branch trunk updated: HDDS-1029. Allow option for force in DeleteContainerCommand. Contributed by Bharat Viswanadham.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 3efa168 HDDS-1029. Allow option for force in DeleteContainerCommand. Contributed by Bharat Viswanadham. 3efa168 is described below commit 3efa168e1f7b4ef7ab741c1fc03d6fc71b653bec Author: Yiqun Lin AuthorDate: Tue Feb 5 10:51:52 2019 +0800 HDDS-1029. Allow option for force in DeleteContainerCommand. Contributed by Bharat Viswanadham. --- .../src/main/proto/DatanodeContainerProtocol.proto | 1 - .../container/common/impl/HddsDispatcher.java | 1 - .../container/common/interfaces/Container.java | 3 +- .../ozone/container/common/interfaces/Handler.java | 4 +- .../DeleteContainerCommandHandler.java | 3 +- .../container/keyvalue/KeyValueContainer.java | 5 +- .../ozone/container/keyvalue/KeyValueHandler.java | 27 ++-- .../keyvalue/helpers/KeyValueContainerUtil.java| 3 +- .../container/ozoneimpl/ContainerController.java | 7 +- .../protocol/commands/DeleteContainerCommand.java | 28 +++- .../proto/StorageContainerDatanodeProtocol.proto | 1 + .../container/keyvalue/TestKeyValueContainer.java | 4 +- .../container/replication/ReplicationManager.java | 2 +- .../common/impl/TestContainerPersistence.java | 4 +- .../commandhandler/TestDeleteContainerHandler.java | 156 +++-- .../container/ozoneimpl/TestOzoneContainer.java| 27 +--- 16 files changed, 174 insertions(+), 102 deletions(-) diff --git a/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto b/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto index 3e4f64d..197bfad 100644 --- a/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto +++ b/hadoop-hdds/common/src/main/proto/DatanodeContainerProtocol.proto @@ -126,7 +126,6 @@ enum Result { PUT_SMALL_FILE_ERROR = 20; GET_SMALL_FILE_ERROR = 21; CLOSED_CONTAINER_IO = 22; - ERROR_CONTAINER_NOT_EMPTY = 23; ERROR_IN_COMPACT_DB = 24; UNCLOSED_CONTAINER_IO = 25; DELETE_ON_OPEN_CONTAINER = 26; diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java index c5c51a3..2f37344 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java @@ -126,7 +126,6 @@ public class HddsDispatcher implements ContainerDispatcher, Auditor { case CONTAINER_UNHEALTHY: case CLOSED_CONTAINER_IO: case DELETE_ON_OPEN_CONTAINER: -case ERROR_CONTAINER_NOT_EMPTY: return true; default: return false; diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Container.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Container.java index 58e3383..89f09fd 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Container.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Container.java @@ -50,10 +50,9 @@ public interface Container extends RwLock { /** * Deletes the container. * - * @param forceDelete - whether this container should be deleted forcibly. * @throws StorageContainerException */ - void delete(boolean forceDelete) throws StorageContainerException; + void delete() throws StorageContainerException; /** * Update the container. diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Handler.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Handler.java index cd93e48..a3bb34b 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Handler.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/Handler.java @@ -151,9 +151,11 @@ public abstract class Handler { * Deletes the given container. * * @param container container to be deleted + * @param force if this is set to true, we delete container without checking + * state of the container. * @throws IOException */ - public abstract void deleteContainer(Container container) + public abstract void deleteContainer(Container container, boolean force) throws IOException; public void setScmID(String scmId) { diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common
[hadoop] branch trunk updated: HDDS-1025. Handle replication of closed containers in DeadNodeHanlder. Contributed by Bharat Viswanadham.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 16195ea HDDS-1025. Handle replication of closed containers in DeadNodeHanlder. Contributed by Bharat Viswanadham. 16195ea is described below commit 16195eaee1b4a7620bc018f48e9c24fc5fc7cc02 Author: Yiqun Lin AuthorDate: Fri Feb 1 11:34:51 2019 +0800 HDDS-1025. Handle replication of closed containers in DeadNodeHanlder. Contributed by Bharat Viswanadham. --- .../hadoop/hdds/scm/node/DeadNodeHandler.java | 28 .../java/org/apache/hadoop/hdds/scm/TestUtils.java | 15 +++ .../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 51 ++ 3 files changed, 77 insertions(+), 17 deletions(-) diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DeadNodeHandler.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DeadNodeHandler.java index 43f0167..8e71399 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DeadNodeHandler.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DeadNodeHandler.java @@ -21,6 +21,7 @@ package org.apache.hadoop.hdds.scm.node; import java.util.Set; import org.apache.hadoop.hdds.protocol.DatanodeDetails; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; import org.apache.hadoop.hdds.scm.container.ContainerException; import org.apache.hadoop.hdds.scm.container.ContainerID; import org.apache.hadoop.hdds.scm.container.ContainerInfo; @@ -81,8 +82,6 @@ public class DeadNodeHandler implements EventHandler { try { final ContainerInfo container = containerManager.getContainer(id); // TODO: For open containers, trigger close on other nodes -// TODO: Check replica count and call replication manager -// on these containers. if (!container.isOpen()) { Set replicas = containerManager .getContainerReplicas(id); @@ -92,6 +91,9 @@ public class DeadNodeHandler implements EventHandler { .ifPresent(replica -> { try { containerManager.removeContainerReplica(id, replica); + ContainerInfo containerInfo = + containerManager.getContainer(id); + replicateIfNeeded(containerInfo, publisher); } catch (ContainerException ex) { LOG.warn("Exception while removing container replica #{} " + "for container #{}.", replica, container, ex); @@ -109,13 +111,21 @@ public class DeadNodeHandler implements EventHandler { */ private void replicateIfNeeded(ContainerInfo container, EventPublisher publisher) throws ContainerNotFoundException { -final int existingReplicas = containerManager -.getContainerReplicas(container.containerID()).size(); -final int expectedReplicas = container.getReplicationFactor().getNumber(); -if (existingReplicas != expectedReplicas) { - publisher.fireEvent(SCMEvents.REPLICATE_CONTAINER, - new ReplicationRequest( - container.getContainerID(), existingReplicas, expectedReplicas)); +// Replicate only closed and Quasi closed containers +if (container.getState() == HddsProtos.LifeCycleState.CLOSED || +container.getState() == HddsProtos.LifeCycleState.QUASI_CLOSED) { + final int existingReplicas = containerManager + .getContainerReplicas(container.containerID()).size(); + final int expectedReplicas = container.getReplicationFactor().getNumber(); + if (existingReplicas != expectedReplicas) { +LOG.debug("Replicate Request fired for container {}, exisiting " + +"replica count {}, expected replica count {}", +container.getContainerID(), existingReplicas, expectedReplicas); +publisher.fireEvent(SCMEvents.REPLICATE_CONTAINER, +new ReplicationRequest( +container.getContainerID(), existingReplicas, +expectedReplicas)); + } } } diff --git a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/TestUtils.java b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/TestUtils.java index e99739c..0f9a5e4 100644 --- a/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/TestUtils.java +++ b/hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/TestUtils.java @@ -446,4 +446,19 @@ public final class TestUtils { id, HddsProtos.LifeCycleEvent.CLOSE); } + + /** + * Move the container to Quaise close state. + * @param containerManager + * @param id + * @throws IOException + */ + public static void quasiCloseContainer(ContainerManager containerManager
[hadoop] branch trunk updated: HDDS-1024. Handle DeleteContainerCommand in the SCMDatanodeProtocolServer. Contributed by Bharat Viswanadham.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new d583cc4 HDDS-1024. Handle DeleteContainerCommand in the SCMDatanodeProtocolServer. Contributed by Bharat Viswanadham. d583cc4 is described below commit d583cc45c69d0363790efbc7b15d339e4492891f Author: Yiqun Lin AuthorDate: Wed Jan 30 13:56:28 2019 +0800 HDDS-1024. Handle DeleteContainerCommand in the SCMDatanodeProtocolServer. Contributed by Bharat Viswanadham. --- .../hdds/scm/server/SCMDatanodeProtocolServer.java | 9 + .../commandhandler/TestDeleteContainerHandler.java | 209 + .../statemachine/commandhandler/package-info.java | 21 +++ 3 files changed, 239 insertions(+) diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java index 06a4a86..3030aa7 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java @@ -58,6 +58,9 @@ import static org.apache.hadoop.hdds.protocol.proto .StorageContainerDatanodeProtocolProtos.SCMCommandProto .Type.deleteBlocksCommand; import static org.apache.hadoop.hdds.protocol.proto +.StorageContainerDatanodeProtocolProtos.SCMCommandProto +.Type.deleteContainerCommand; +import static org.apache.hadoop.hdds.protocol.proto .StorageContainerDatanodeProtocolProtos.SCMCommandProto.Type .replicateContainerCommand; import static org.apache.hadoop.hdds.protocol.proto @@ -87,6 +90,7 @@ import org.apache.hadoop.ozone.audit.SCMAction; import org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol; import org.apache.hadoop.ozone.protocol.commands.CloseContainerCommand; import org.apache.hadoop.ozone.protocol.commands.DeleteBlocksCommand; +import org.apache.hadoop.ozone.protocol.commands.DeleteContainerCommand; import org.apache.hadoop.ozone.protocol.commands.RegisteredCommand; import org.apache.hadoop.ozone.protocol.commands.ReplicateContainerCommand; import org.apache.hadoop.ozone.protocol.commands.SCMCommand; @@ -335,6 +339,11 @@ public class SCMDatanodeProtocolServer implements .setCloseContainerCommandProto( ((CloseContainerCommand) cmd).getProto()) .build(); +case deleteContainerCommand: + return builder.setCommandType(deleteContainerCommand) + .setDeleteContainerCommandProto( + ((DeleteContainerCommand) cmd).getProto()) + .build(); case replicateContainerCommand: return builder .setCommandType(replicateContainerCommand) diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestDeleteContainerHandler.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestDeleteContainerHandler.java new file mode 100644 index 000..232ab0ac --- /dev/null +++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestDeleteContainerHandler.java @@ -0,0 +1,209 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ + +package org.apache.hadoop.ozone.container.common.statemachine.commandhandler; + +import org.apache.hadoop.hdds.client.ReplicationFactor; +import org.apache.hadoop.hdds.client.ReplicationType; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; +import org.apache.hadoop.hdds.protocol.DatanodeDetails; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; +import org.apache.hadoop.hdds.scm.container.ContainerID; +import org.apache.hadoop.hdds.scm.container.ContainerInfo; +import org.apache.hadoop.hdds.scm.pipeline.Pipeline; +import org.apache.hadoop.ozone.HddsDatanodeService; +import org.apache.hadoop.ozone.MiniOzoneCluster; +import org.apache.hadoop.ozone.client.
[hadoop] branch trunk updated: HDDS-1022. Add cmd type in getCommandResponse in SCMDatanodeProtocolServer. Contributed by Bharat Viswanadham.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 2d06112 HDDS-1022. Add cmd type in getCommandResponse in SCMDatanodeProtocolServer. Contributed by Bharat Viswanadham. 2d06112 is described below commit 2d06112b74c1f797db1f2b4884c1de4485b15dfa Author: Yiqun Lin AuthorDate: Tue Jan 29 11:26:41 2019 +0800 HDDS-1022. Add cmd type in getCommandResponse in SCMDatanodeProtocolServer. Contributed by Bharat Viswanadham. --- .../org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java index 6a3552e..06a4a86 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeProtocolServer.java @@ -342,7 +342,8 @@ public class SCMDatanodeProtocolServer implements ((ReplicateContainerCommand)cmd).getProto()) .build(); default: - throw new IllegalArgumentException("Not implemented"); + throw new IllegalArgumentException("Scm command " + + cmd.getType().toString() + " is not implemented"); } } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HDDS-974. Add getServiceAddress method to ServiceInfo and use it in TestOzoneShell. Contributed by Doroszlai, Attila.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/trunk by this push: new 8326450 HDDS-974. Add getServiceAddress method to ServiceInfo and use it in TestOzoneShell. Contributed by Doroszlai, Attila. 8326450 is described below commit 8326450bca1b15383bf1c1a3b4e86df1ca027495 Author: Yiqun Lin AuthorDate: Mon Jan 28 17:03:12 2019 +0800 HDDS-974. Add getServiceAddress method to ServiceInfo and use it in TestOzoneShell. Contributed by Doroszlai, Attila. --- .../apache/hadoop/ozone/client/rpc/RpcClient.java | 5 +- .../hadoop/ozone/om/helpers/ServiceInfo.java | 14 - .../ozone/ozShell/TestOzoneDatanodeShell.java | 34 +-- .../hadoop/ozone/ozShell/TestOzoneShell.java | 66 -- 4 files changed, 40 insertions(+), 79 deletions(-) diff --git a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java index 538f69b..8f398e9 100644 --- a/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java +++ b/hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java @@ -80,7 +80,6 @@ import org.apache.ratis.protocol.ClientId; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import javax.ws.rs.HEAD; import java.io.IOException; import java.net.InetSocketAddress; import java.util.*; @@ -206,8 +205,8 @@ public class RpcClient implements ClientProtocol { ServiceInfo scmInfo = services.stream().filter( a -> a.getNodeType().equals(HddsProtos.NodeType.SCM)) .collect(Collectors.toList()).get(0); -return NetUtils.createSocketAddr(scmInfo.getHostname()+ ":" + -scmInfo.getPort(ServicePort.Type.RPC)); +return NetUtils.createSocketAddr( +scmInfo.getServiceAddress(ServicePort.Type.RPC)); } @Override diff --git a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/ServiceInfo.java b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/ServiceInfo.java index 9b03aef..fb49644 100644 --- a/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/ServiceInfo.java +++ b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/ServiceInfo.java @@ -109,11 +109,11 @@ public final class ServiceInfo { } /** - * Returns the port for given type, null if the service doesn't support - * the type. + * Returns the port for given type. * * @param type the type of port. * ex: RPC, HTTP, HTTPS, etc.. + * @throws NullPointerException if the service doesn't support the given type */ @JsonIgnore public int getPort(ServicePort.Type type) { @@ -121,6 +121,16 @@ public final class ServiceInfo { } /** + * Returns the address of the service (hostname with port of the given type). + * @param portType the type of port, eg. RPC, HTTP, etc. + * @return service address (hostname with port of the given type) + */ + @JsonIgnore + public String getServiceAddress(ServicePort.Type portType) { +return hostname + ":" + getPort(portType); + } + + /** * Converts {@link ServiceInfo} to OzoneManagerProtocolProtos.ServiceInfo. * * @return OzoneManagerProtocolProtos.ServiceInfo diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestOzoneDatanodeShell.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestOzoneDatanodeShell.java index a45dee8..9ba8529 100644 --- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestOzoneDatanodeShell.java +++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestOzoneDatanodeShell.java @@ -21,25 +21,19 @@ import com.google.common.base.Strings; import java.io.ByteArrayOutputStream; import java.io.File; -import java.io.IOException; import java.io.PrintStream; -import java.util.ArrayList; import java.util.Arrays; import java.util.Collection; import java.util.List; -import java.util.stream.Collectors; import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.hdds.cli.MissingSubcommandException; import org.apache.hadoop.hdds.client.ReplicationFactor; import org.apache.hadoop.hdds.conf.OzoneConfiguration; -import org.apache.hadoop.hdds.protocol.proto.HddsProtos; import org.apache.hadoop.ozone.HddsDatanodeService; import org.apache.hadoop.ozone.MiniOzoneCluster; -import org.apache.hadoop.ozone.client.protocol.ClientProtocol; import org.apache.hadoop.ozone.client.rest.RestClient; import org.apache.hadoop.ozone.client.rpc.RpcClient; -import org.apache.hadoop.ozone.om.helpers.ServiceInfo; import org.apache.hadoop.test.GenericTestUtils; import org.junit.After; impor
[hadoop] branch HDFS-13891 updated: HDFS-14209. RBF: setQuota() through router is working for only the mount Points under the Source column in MountTable. Contributed by Shubham Dewan.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch HDFS-13891 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/HDFS-13891 by this push: new 7fe0b06 HDFS-14209. RBF: setQuota() through router is working for only the mount Points under the Source column in MountTable. Contributed by Shubham Dewan. 7fe0b06 is described below commit 7fe0b0684c67d2a766eebce663ac145e63fe4acb Author: Yiqun Lin AuthorDate: Wed Jan 23 22:59:43 2019 +0800 HDFS-14209. RBF: setQuota() through router is working for only the mount Points under the Source column in MountTable. Contributed by Shubham Dewan. --- .../hdfs/server/federation/router/Quota.java | 7 - .../server/federation/router/TestRouterQuota.java | 32 +- 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java index cfb538f..a6f5bab 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java @@ -216,6 +216,11 @@ public class Quota { locations.addAll(rpcServer.getLocationsForPath(childPath, true, false)); } } -return locations; +if (locations.size() >= 1) { + return locations; +} else { + locations.addAll(rpcServer.getLocationsForPath(path, true, false)); + return locations; +} } } diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterQuota.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterQuota.java index 656b401..034023c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterQuota.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterQuota.java @@ -755,4 +755,34 @@ public class TestRouterQuota { assertEquals(HdfsConstants.QUOTA_RESET, subClusterQuota.getQuota()); assertEquals(HdfsConstants.QUOTA_RESET, subClusterQuota.getSpaceQuota()); } -} \ No newline at end of file + + @Test + public void testSetQuotaNotMountTable() throws Exception { +long nsQuota = 5; +long ssQuota = 100; +final FileSystem nnFs1 = nnContext1.getFileSystem(); + +// setQuota should run for any directory +MountTable mountTable1 = MountTable.newInstance("/setquotanmt", +Collections.singletonMap("ns0", "/testdir16")); + +addMountTable(mountTable1); + +// Add a directory not present in mount table. +nnFs1.mkdirs(new Path("/testdir16/testdir17")); + +routerContext.getRouter().getRpcServer().setQuota("/setquotanmt/testdir17", +nsQuota, ssQuota, null); + +RouterQuotaUpdateService updateService = routerContext.getRouter() +.getQuotaCacheUpdateService(); +// ensure setQuota RPC call was invoked +updateService.periodicInvoke(); + +ClientProtocol client1 = nnContext1.getClient().getNamenode(); +final QuotaUsage quota1 = client1.getQuotaUsage("/testdir16/testdir17"); + +assertEquals(nsQuota, quota1.getQuota()); +assertEquals(ssQuota, quota1.getSpaceQuota()); + } +} - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch HDFS-13891 updated: HDFS-14206. RBF: Cleanup quota modules. Contributed by Inigo Goiri.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch HDFS-13891 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/HDFS-13891 by this push: new c9a6545 HDFS-14206. RBF: Cleanup quota modules. Contributed by Inigo Goiri. c9a6545 is described below commit c9a65456fc64b16289d62d72a22cf7890605f024 Author: Yiqun Lin AuthorDate: Tue Jan 15 14:21:33 2019 +0800 HDFS-14206. RBF: Cleanup quota modules. Contributed by Inigo Goiri. --- .../hdfs/server/federation/router/Quota.java | 6 ++-- .../federation/router/RouterClientProtocol.java| 22 +++--- .../federation/router/RouterQuotaManager.java | 2 +- .../router/RouterQuotaUpdateService.java | 6 ++-- .../server/federation/router/RouterQuotaUsage.java | 35 -- 5 files changed, 38 insertions(+), 33 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java index 5d0309f..cfb538f 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java @@ -163,7 +163,7 @@ public class Quota { long ssCount = 0; long nsQuota = HdfsConstants.QUOTA_RESET; long ssQuota = HdfsConstants.QUOTA_RESET; -boolean hasQuotaUnSet = false; +boolean hasQuotaUnset = false; for (Map.Entry entry : results.entrySet()) { RemoteLocation loc = entry.getKey(); @@ -172,7 +172,7 @@ public class Quota { // If quota is not set in real FileSystem, the usage // value will return -1. if (usage.getQuota() == -1 && usage.getSpaceQuota() == -1) { - hasQuotaUnSet = true; + hasQuotaUnset = true; } nsQuota = usage.getQuota(); ssQuota = usage.getSpaceQuota(); @@ -189,7 +189,7 @@ public class Quota { QuotaUsage.Builder builder = new QuotaUsage.Builder() .fileAndDirectoryCount(nsCount).spaceConsumed(ssCount); -if (hasQuotaUnSet) { +if (hasQuotaUnset) { builder.quota(HdfsConstants.QUOTA_RESET) .spaceQuota(HdfsConstants.QUOTA_RESET); } else { diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java index 894f5be..0962c0d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java @@ -20,7 +20,7 @@ package org.apache.hadoop.hdfs.server.federation.router; import static org.apache.hadoop.hdfs.server.federation.router.FederationUtil.updateMountPointStatus; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.crypto.CryptoProtocolVersion; -import org.apache.hadoop.fs.BatchedRemoteIterator; +import org.apache.hadoop.fs.BatchedRemoteIterator.BatchedEntries; import org.apache.hadoop.fs.CacheFlag; import org.apache.hadoop.fs.ContentSummary; import org.apache.hadoop.fs.CreateFlag; @@ -1140,7 +1140,7 @@ public class RouterClientProtocol implements ClientProtocol { } @Override - public BatchedRemoteIterator.BatchedEntries listCacheDirectives( + public BatchedEntries listCacheDirectives( long prevId, CacheDirectiveInfo filter) throws IOException { rpcServer.checkOperation(NameNode.OperationCategory.READ, false); return null; @@ -1162,7 +1162,7 @@ public class RouterClientProtocol implements ClientProtocol { } @Override - public BatchedRemoteIterator.BatchedEntries listCachePools(String prevKey) + public BatchedEntries listCachePools(String prevKey) throws IOException { rpcServer.checkOperation(NameNode.OperationCategory.READ, false); return null; @@ -1273,7 +1273,7 @@ public class RouterClientProtocol implements ClientProtocol { } @Override - public BatchedRemoteIterator.BatchedEntries listEncryptionZones(long prevId) + public BatchedEntries listEncryptionZones(long prevId) throws IOException { rpcServer.checkOperation(NameNode.OperationCategory.READ, false); return null; @@ -1286,7 +1286,7 @@ public class RouterClientProtocol implements ClientProtocol { } @Override - public BatchedRemoteIterator.BatchedEntries listReencryptionStatus( + public BatchedEntries listReencryptionStatus( long prevId) throws IOException { rpcServer.checkOperation(NameNode.OperationCategory.READ, false); ret
[hadoop] branch HDFS-13891 updated: HDFS-14150. RBF: Quotas of the sub-cluster should be removed when removing the mount point. Contributed by Takanobu Asanuma.
This is an automated email from the ASF dual-hosted git repository. yqlin pushed a commit to branch HDFS-13891 in repository https://gitbox.apache.org/repos/asf/hadoop.git The following commit(s) were added to refs/heads/HDFS-13891 by this push: new 8c245e7 HDFS-14150. RBF: Quotas of the sub-cluster should be removed when removing the mount point. Contributed by Takanobu Asanuma. 8c245e7 is described below commit 8c245e75b4f8020cd3353c0263ea83704d1096c4 Author: Yiqun Lin AuthorDate: Wed Jan 9 17:18:43 2019 +0800 HDFS-14150. RBF: Quotas of the sub-cluster should be removed when removing the mount point. Contributed by Takanobu Asanuma. --- .../federation/router/RouterAdminServer.java | 23 +++ .../src/main/resources/hdfs-rbf-default.xml| 4 +- .../src/site/markdown/HDFSRouterFederation.md | 4 +- .../server/federation/router/TestRouterQuota.java | 48 +- 4 files changed, 67 insertions(+), 12 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java index 5bb7751..18c19e0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAdminServer.java @@ -250,23 +250,25 @@ public class RouterAdminServer extends AbstractService MountTable mountTable = request.getEntry(); if (mountTable != null && router.isQuotaEnabled()) { - synchronizeQuota(mountTable); + synchronizeQuota(mountTable.getSourcePath(), + mountTable.getQuota().getQuota(), + mountTable.getQuota().getSpaceQuota()); } return response; } /** * Synchronize the quota value across mount table and subclusters. - * @param mountTable Quota set in given mount table. + * @param path Source path in given mount table. + * @param nsQuota Name quota definition in given mount table. + * @param ssQuota Space quota definition in given mount table. * @throws IOException */ - private void synchronizeQuota(MountTable mountTable) throws IOException { -String path = mountTable.getSourcePath(); -long nsQuota = mountTable.getQuota().getQuota(); -long ssQuota = mountTable.getQuota().getSpaceQuota(); - -if (nsQuota != HdfsConstants.QUOTA_DONT_SET -|| ssQuota != HdfsConstants.QUOTA_DONT_SET) { + private void synchronizeQuota(String path, long nsQuota, long ssQuota) + throws IOException { +if (router.isQuotaEnabled() && +(nsQuota != HdfsConstants.QUOTA_DONT_SET +|| ssQuota != HdfsConstants.QUOTA_DONT_SET)) { HdfsFileStatus ret = this.router.getRpcServer().getFileInfo(path); if (ret != null) { this.router.getRpcServer().getQuotaModule().setQuota(path, nsQuota, @@ -278,6 +280,9 @@ public class RouterAdminServer extends AbstractService @Override public RemoveMountTableEntryResponse removeMountTableEntry( RemoveMountTableEntryRequest request) throws IOException { +// clear sub-cluster's quota definition +synchronizeQuota(request.getSrcPath(), HdfsConstants.QUOTA_RESET, +HdfsConstants.QUOTA_RESET); return getMountTableStore().removeMountTableEntry(request); } diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/resources/hdfs-rbf-default.xml b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/resources/hdfs-rbf-default.xml index 72f6c2f..20ae778 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/resources/hdfs-rbf-default.xml +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/resources/hdfs-rbf-default.xml @@ -447,7 +447,9 @@ dfs.federation.router.quota.enable false - Set to true to enable quota system in Router. + Set to true to enable quota system in Router. When it's enabled, setting + or clearing sub-cluster's quota directly is not recommended since Router + Admin server will override sub-cluster's quota with global quota. diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md index adc4383..959cd63 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md @@ -143,6 +143,8 @@ For performance reasons, the Router caches the quota usage and updates it period will be used for quota-verification during each WRITE RPC call invoked in RouterRPCSever. See [HDFS Quotas Guide](../hadoop-hdfs/HdfsQuotaAdminGuide.html) for the quota detail. +Note: When global quota is enabled, setting or clearing sub-cluster&
hadoop git commit: HDFS-13946. Log longest FSN write/read lock held stack trace.
Repository: hadoop Updated Branches: refs/heads/trunk 763e96ea2 -> feb2664ac HDFS-13946. Log longest FSN write/read lock held stack trace. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/feb2664a Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/feb2664a Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/feb2664a Branch: refs/heads/trunk Commit: feb2664ac4b246ca87fc4997a941190f00026dff Parents: 763e96e Author: Yiqun Lin Authored: Sat Dec 22 23:09:59 2018 +0800 Committer: Yiqun Lin Committed: Sat Dec 22 23:09:59 2018 +0800 -- .../apache/hadoop/log/LogThrottlingHelper.java | 16 .../hadoop/log/TestLogThrottlingHelper.java | 3 + .../hdfs/server/namenode/FSNamesystemLock.java | 93 .../server/namenode/TestFSNamesystemLock.java | 41 +++-- 4 files changed, 126 insertions(+), 27 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/feb2664a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/log/LogThrottlingHelper.java -- diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/log/LogThrottlingHelper.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/log/LogThrottlingHelper.java index 41bee04..848f123 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/log/LogThrottlingHelper.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/log/LogThrottlingHelper.java @@ -273,6 +273,22 @@ public class LogThrottlingHelper { } /** + * Return the summary information for given index. + * + * @param recorderName The name of the recorder. + * @param idx The index value. + * @return The summary information. + */ + public SummaryStatistics getCurrentStats(String recorderName, int idx) { +LoggingAction currentLog = currentLogs.get(recorderName); +if (currentLog != null) { + return currentLog.getStats(idx); +} + +return null; + } + + /** * A standard log action which keeps track of all of the values which have * been logged. This is also used for internal bookkeeping via its private * fields and methods; it will maintain whether or not it is ready to be http://git-wip-us.apache.org/repos/asf/hadoop/blob/feb2664a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogThrottlingHelper.java -- diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogThrottlingHelper.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogThrottlingHelper.java index a675d0a..d0eeea3 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogThrottlingHelper.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/log/TestLogThrottlingHelper.java @@ -167,6 +167,9 @@ public class TestLogThrottlingHelper { assertEquals(2.0, bar.getStats(0).getMean(), 0.01); assertEquals(3.0, baz.getStats(0).getMean(), 0.01); assertEquals(3.0, baz.getStats(1).getMean(), 0.01); + +assertEquals(2.0, helper.getCurrentStats("bar", 0).getMax(), 0); +assertEquals(3.0, helper.getCurrentStats("baz", 0).getMax(), 0); } } http://git-wip-us.apache.org/repos/asf/hadoop/blob/feb2664a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java index 7c28465..ebf5178 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java @@ -21,10 +21,12 @@ package org.apache.hadoop.hdfs.server.namenode; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.atomic.AtomicReference; import java.util.concurrent.locks.Condition; import java.util.concurrent.locks.ReentrantReadWriteLock; import com.google.common.annotations.VisibleForTesting; +import org.apache.commons.math3.stat.descriptive.SummaryStatistics; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.log.LogThrottlingHelper; import org.apache.hadoop.metrics2.lib.MutableRatesWithAggregation; @@ -97,8 +99,18 @@ class FSNamesys
[1/2] hadoop git commit: HDFS-13443. RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries. Contributed by Mohammad Arshad.
Repository: hadoop Updated Branches: refs/heads/HDFS-13891 7d7e88c30 -> c9ebaf2d3 http://git-wip-us.apache.org/repos/asf/hadoop/blob/c9ebaf2d/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterMountTableCacheRefresh.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterMountTableCacheRefresh.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterMountTableCacheRefresh.java new file mode 100644 index 000..c90e614 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterMountTableCacheRefresh.java @@ -0,0 +1,396 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hdfs.server.federation.router; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.net.InetSocketAddress; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import org.apache.curator.test.TestingServer; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.CommonConfigurationKeys; +import org.apache.hadoop.hdfs.server.federation.FederationTestUtils; +import org.apache.hadoop.hdfs.server.federation.MiniRouterDFSCluster; +import org.apache.hadoop.hdfs.server.federation.MiniRouterDFSCluster.RouterContext; +import org.apache.hadoop.hdfs.server.federation.RouterConfigBuilder; +import org.apache.hadoop.hdfs.server.federation.resolver.FileSubclusterResolver; +import org.apache.hadoop.hdfs.server.federation.resolver.MountTableManager; +import org.apache.hadoop.hdfs.server.federation.store.RouterStore; +import org.apache.hadoop.hdfs.server.federation.store.protocol.AddMountTableEntryRequest; +import org.apache.hadoop.hdfs.server.federation.store.protocol.AddMountTableEntryResponse; +import org.apache.hadoop.hdfs.server.federation.store.protocol.GetMountTableEntriesRequest; +import org.apache.hadoop.hdfs.server.federation.store.protocol.RefreshMountTableEntriesRequest; +import org.apache.hadoop.hdfs.server.federation.store.protocol.RefreshMountTableEntriesResponse; +import org.apache.hadoop.hdfs.server.federation.store.protocol.RemoveMountTableEntryRequest; +import org.apache.hadoop.hdfs.server.federation.store.protocol.RemoveMountTableEntryResponse; +import org.apache.hadoop.hdfs.server.federation.store.protocol.UpdateMountTableEntryRequest; +import org.apache.hadoop.hdfs.server.federation.store.protocol.UpdateMountTableEntryResponse; +import org.apache.hadoop.hdfs.server.federation.store.records.MountTable; +import org.apache.hadoop.service.Service.STATE; +import org.apache.hadoop.test.GenericTestUtils; +import org.apache.hadoop.util.Time; +import org.junit.After; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; + +/** + * This test class verifies that mount table cache is updated on all the routers + * when MountTableRefreshService is enabled and there is a change in mount table + * entries. + */ +public class TestRouterMountTableCacheRefresh { + private static TestingServer curatorTestingServer; + private static MiniRouterDFSCluster cluster; + private static RouterContext routerContext; + private static MountTableManager mountTableManager; + + @BeforeClass + public static void setUp() throws Exception { +curatorTestingServer = new TestingServer(); +curatorTestingServer.start(); +final String connectString = curatorTestingServer.getConnectString(); +int numNameservices = 2; +cluster = new MiniRouterDFSCluster(false, numNameservices); +Configuration conf = new RouterConfigBuilder().refreshCache().admin().rpc() +.heartbeat().build(); +conf.setClass(RBFConfigKeys.FEDERATION_FILE_RESOLVER_CLIENT_CLASS, +RBFConfigKeys.FE
[2/2] hadoop git commit: HDFS-13443. RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries. Contributed by Mohammad Arshad.
HDFS-13443. RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries. Contributed by Mohammad Arshad. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c9ebaf2d Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c9ebaf2d Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c9ebaf2d Branch: refs/heads/HDFS-13891 Commit: c9ebaf2d3f8204c31ff9796b17af71fc2be9a6bd Parents: 7d7e88c Author: Yiqun Lin Authored: Wed Dec 19 11:40:00 2018 +0800 Committer: Yiqun Lin Committed: Wed Dec 19 11:40:00 2018 +0800 -- ...uterAdminProtocolServerSideTranslatorPB.java | 23 ++ .../RouterAdminProtocolTranslatorPB.java| 21 + .../federation/resolver/MountTableManager.java | 16 + .../router/MountTableRefresherService.java | 289 ++ .../router/MountTableRefresherThread.java | 96 + .../server/federation/router/RBFConfigKeys.java | 25 ++ .../hdfs/server/federation/router/Router.java | 53 ++- .../federation/router/RouterAdminServer.java| 28 +- .../router/RouterHeartbeatService.java | 5 + .../federation/store/MountTableStore.java | 24 ++ .../federation/store/StateStoreUtils.java | 26 ++ .../store/impl/MountTableStoreImpl.java | 18 + .../RefreshMountTableEntriesRequest.java| 34 ++ .../RefreshMountTableEntriesResponse.java | 44 +++ .../RefreshMountTableEntriesRequestPBImpl.java | 67 .../RefreshMountTableEntriesResponsePBImpl.java | 74 .../federation/store/records/RouterState.java | 4 + .../records/impl/pb/RouterStatePBImpl.java | 10 + .../hdfs/tools/federation/RouterAdmin.java | 33 +- .../src/main/proto/FederationProtocol.proto | 8 + .../src/main/proto/RouterProtocol.proto | 5 + .../src/main/resources/hdfs-rbf-default.xml | 34 ++ .../src/site/markdown/HDFSRouterFederation.md | 9 + .../server/federation/FederationTestUtils.java | 27 ++ .../server/federation/RouterConfigBuilder.java | 12 + .../federation/router/TestRouterAdminCLI.java | 25 +- .../TestRouterMountTableCacheRefresh.java | 396 +++ .../src/site/markdown/HDFSCommands.md | 2 + 28 files changed, 1402 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/c9ebaf2d/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/protocolPB/RouterAdminProtocolServerSideTranslatorPB.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/protocolPB/RouterAdminProtocolServerSideTranslatorPB.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/protocolPB/RouterAdminProtocolServerSideTranslatorPB.java index 6341ebd..a31c46d 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/protocolPB/RouterAdminProtocolServerSideTranslatorPB.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/protocolPB/RouterAdminProtocolServerSideTranslatorPB.java @@ -37,6 +37,8 @@ import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProt import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.GetSafeModeResponseProto; import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.LeaveSafeModeRequestProto; import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.LeaveSafeModeResponseProto; +import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RefreshMountTableEntriesRequestProto; +import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RefreshMountTableEntriesResponseProto; import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RemoveMountTableEntryRequestProto; import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.RemoveMountTableEntryResponseProto; import org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos.UpdateMountTableEntryRequestProto; @@ -58,6 +60,8 @@ import org.apache.hadoop.hdfs.server.federation.store.protocol.GetSafeModeReques import org.apache.hadoop.hdfs.server.federation.store.protocol.GetSafeModeResponse; import org.apache.hadoop.hdfs.server.federation.store.protocol.LeaveSafeModeRequest; import org.apache.hadoop.hdfs.server.federation.store.protocol.LeaveSafeModeResponse; +import org.apache.hadoop.hdfs.server.federation.store.protocol.RefreshMountTableEntriesRequest; +import org.apache.hadoop.hdfs.server.federation.store.protocol.RefreshMountTableEntriesResponse; import org.apache.hadoop.hdfs.server.federation.store.protocol.Remove
hadoop git commit: HDFS-13869. RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics. Contributed by Ranith Sardar.
Repository: hadoop Updated Branches: refs/heads/HDFS-13891 47efde24f -> a6f61f9b3 HDFS-13869. RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics. Contributed by Ranith Sardar. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a6f61f9b Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a6f61f9b Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a6f61f9b Branch: refs/heads/HDFS-13891 Commit: a6f61f9b379d5ffd2b7d6448e0644e285ede920a Parents: 47efde2 Author: Yiqun Lin Authored: Mon Dec 17 12:35:07 2018 +0800 Committer: Yiqun Lin Committed: Mon Dec 17 12:35:07 2018 +0800 -- .../federation/metrics/NamenodeBeanMetrics.java | 149 --- .../hdfs/server/federation/router/Router.java | 8 +- .../server/federation/router/TestRouter.java| 14 ++ 3 files changed, 147 insertions(+), 24 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/a6f61f9b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/NamenodeBeanMetrics.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/NamenodeBeanMetrics.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/NamenodeBeanMetrics.java index 64df10c..25ec27c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/NamenodeBeanMetrics.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/NamenodeBeanMetrics.java @@ -168,8 +168,12 @@ public class NamenodeBeanMetrics } } - private FederationMetrics getFederationMetrics() { -return this.router.getMetrics(); + private FederationMetrics getFederationMetrics() throws IOException { +FederationMetrics metrics = getRouter().getMetrics(); +if (metrics == null) { + throw new IOException("Federated metrics is not initialized"); +} +return metrics; } / @@ -188,22 +192,42 @@ public class NamenodeBeanMetrics @Override public long getUsed() { -return getFederationMetrics().getUsedCapacity(); +try { + return getFederationMetrics().getUsedCapacity(); +} catch (IOException e) { + LOG.debug("Failed to get the used capacity", e.getMessage()); +} +return 0; } @Override public long getFree() { -return getFederationMetrics().getRemainingCapacity(); +try { + return getFederationMetrics().getRemainingCapacity(); +} catch (IOException e) { + LOG.debug("Failed to get remaining capacity", e.getMessage()); +} +return 0; } @Override public long getTotal() { -return getFederationMetrics().getTotalCapacity(); +try { + return getFederationMetrics().getTotalCapacity(); +} catch (IOException e) { + LOG.debug("Failed to Get total capacity", e.getMessage()); +} +return 0; } @Override public long getProvidedCapacity() { -return getFederationMetrics().getProvidedSpace(); +try { + return getFederationMetrics().getProvidedSpace(); +} catch (IOException e) { + LOG.debug("Failed to get provided capacity", e.getMessage()); +} +return 0; } @Override @@ -261,39 +285,79 @@ public class NamenodeBeanMetrics @Override public long getTotalBlocks() { -return getFederationMetrics().getNumBlocks(); +try { + return getFederationMetrics().getNumBlocks(); +} catch (IOException e) { + LOG.debug("Failed to get number of blocks", e.getMessage()); +} +return 0; } @Override public long getNumberOfMissingBlocks() { -return getFederationMetrics().getNumOfMissingBlocks(); +try { + return getFederationMetrics().getNumOfMissingBlocks(); +} catch (IOException e) { + LOG.debug("Failed to get number of missing blocks", e.getMessage()); +} +return 0; } @Override @Deprecated public long getPendingReplicationBlocks() { -return getFederationMetrics().getNumOfBlocksPendingReplication(); +try { + return getFederationMetrics().getNumOfBlocksPendingReplication(); +} catch (IOException e) { + LOG.debug("Failed to get number of blocks pending replica", + e.getMessage()); +} +return 0; } @Override public long getPendingReconstructionBlocks() { -return getFederationMetrics().getNumOfBlocksPendingReplication(); +try { + return getFederationMetrics().getNumOfBlocksPendingReplication(); +} catch (IOException e) { + LOG.debug("Failed to get number of blocks pending replica", +
hadoop git commit: HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui.
Repository: hadoop Updated Branches: refs/heads/HDFS-13891 4df886a64 -> 9c1594708 HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9c159470 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9c159470 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9c159470 Branch: refs/heads/HDFS-13891 Commit: 9c1594708913403784283a29b94f260448969615 Parents: 4df886a Author: Yiqun Lin Authored: Wed Dec 5 11:44:38 2018 +0800 Committer: Yiqun Lin Committed: Wed Dec 5 11:44:38 2018 +0800 -- .../federation/router/ConnectionManager.java| 20 --- .../federation/router/ConnectionPool.java | 14 - .../server/federation/router/RBFConfigKeys.java | 5 ++ .../src/main/resources/hdfs-rbf-default.xml | 8 +++ .../router/TestConnectionManager.java | 55 +--- 5 files changed, 85 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/9c159470/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java index fa2bf94..745 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java @@ -49,10 +49,6 @@ public class ConnectionManager { private static final Logger LOG = LoggerFactory.getLogger(ConnectionManager.class); - /** Minimum amount of active connections: 50%. */ - protected static final float MIN_ACTIVE_RATIO = 0.5f; - - /** Configuration for the connection manager, pool and sockets. */ private final Configuration conf; @@ -60,6 +56,8 @@ public class ConnectionManager { private final int minSize = 1; /** Max number of connections per user + nn. */ private final int maxSize; + /** Min ratio of active connections per user + nn. */ + private final float minActiveRatio; /** How often we close a pool for a particular user + nn. */ private final long poolCleanupPeriodMs; @@ -96,10 +94,13 @@ public class ConnectionManager { public ConnectionManager(Configuration config) { this.conf = config; -// Configure minimum and maximum connection pools +// Configure minimum, maximum and active connection pools this.maxSize = this.conf.getInt( RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_POOL_SIZE, RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_POOL_SIZE_DEFAULT); +this.minActiveRatio = this.conf.getFloat( +RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_MIN_ACTIVE_RATIO, +RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_MIN_ACTIVE_RATIO_DEFAULT); // Map with the connections indexed by UGI and Namenode this.pools = new HashMap<>(); @@ -203,7 +204,8 @@ public class ConnectionManager { pool = this.pools.get(connectionId); if (pool == null) { pool = new ConnectionPool( - this.conf, nnAddress, ugi, this.minSize, this.maxSize, protocol); + this.conf, nnAddress, ugi, this.minSize, this.maxSize, + this.minActiveRatio, protocol); this.pools.put(connectionId, pool); } } finally { @@ -326,8 +328,9 @@ public class ConnectionManager { long timeSinceLastActive = Time.now() - pool.getLastActiveTime(); int total = pool.getNumConnections(); int active = pool.getNumActiveConnections(); + float poolMinActiveRatio = pool.getMinActiveRatio(); if (timeSinceLastActive > connectionCleanupPeriodMs || - active < MIN_ACTIVE_RATIO * total) { + active < poolMinActiveRatio * total) { // Remove and close 1 connection List conns = pool.removeConnections(1); for (ConnectionContext conn : conns) { @@ -412,8 +415,9 @@ public class ConnectionManager { try { int total = pool.getNumConnections(); int active = pool.getNumActiveConnections(); +float poolMinActiveRatio = pool.getMinActiveRatio(); if (pool.getNumConnections() < pool.getMaxSize() && -active >= MIN_ACTIVE_RATIO * total) { +active >= poolMinActiveRatio * total) { ConnectionContext conn = pool.newConnection(); pool.addConnection(conn); } else { ht
hadoop git commit: Revert "HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui."
Repository: hadoop Updated Branches: refs/heads/HDFS-13891 7c0d6f65f -> 4df886a64 Revert "HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui." This reverts commit 7c0d6f65fde12ead91ed7c706521ad1d3dc995f8. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4df886a6 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4df886a6 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4df886a6 Branch: refs/heads/HDFS-13891 Commit: 4df886a64d7595000760556798a33cf017fffb07 Parents: 7c0d6f6 Author: Yiqun Lin Authored: Tue Dec 4 22:16:00 2018 +0800 Committer: Yiqun Lin Committed: Tue Dec 4 22:16:00 2018 +0800 -- .../federation/router/ConnectionManager.java| 20 +++- .../federation/router/ConnectionPool.java | 14 +- .../server/federation/router/RBFConfigKeys.java | 5 -- .../src/main/resources/hdfs-rbf-default.xml | 8 --- .../router/TestConnectionManager.java | 51 +++- 5 files changed, 15 insertions(+), 83 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/4df886a6/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java index 745..fa2bf94 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java @@ -49,6 +49,10 @@ public class ConnectionManager { private static final Logger LOG = LoggerFactory.getLogger(ConnectionManager.class); + /** Minimum amount of active connections: 50%. */ + protected static final float MIN_ACTIVE_RATIO = 0.5f; + + /** Configuration for the connection manager, pool and sockets. */ private final Configuration conf; @@ -56,8 +60,6 @@ public class ConnectionManager { private final int minSize = 1; /** Max number of connections per user + nn. */ private final int maxSize; - /** Min ratio of active connections per user + nn. */ - private final float minActiveRatio; /** How often we close a pool for a particular user + nn. */ private final long poolCleanupPeriodMs; @@ -94,13 +96,10 @@ public class ConnectionManager { public ConnectionManager(Configuration config) { this.conf = config; -// Configure minimum, maximum and active connection pools +// Configure minimum and maximum connection pools this.maxSize = this.conf.getInt( RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_POOL_SIZE, RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_POOL_SIZE_DEFAULT); -this.minActiveRatio = this.conf.getFloat( -RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_MIN_ACTIVE_RATIO, -RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_MIN_ACTIVE_RATIO_DEFAULT); // Map with the connections indexed by UGI and Namenode this.pools = new HashMap<>(); @@ -204,8 +203,7 @@ public class ConnectionManager { pool = this.pools.get(connectionId); if (pool == null) { pool = new ConnectionPool( - this.conf, nnAddress, ugi, this.minSize, this.maxSize, - this.minActiveRatio, protocol); + this.conf, nnAddress, ugi, this.minSize, this.maxSize, protocol); this.pools.put(connectionId, pool); } } finally { @@ -328,9 +326,8 @@ public class ConnectionManager { long timeSinceLastActive = Time.now() - pool.getLastActiveTime(); int total = pool.getNumConnections(); int active = pool.getNumActiveConnections(); - float poolMinActiveRatio = pool.getMinActiveRatio(); if (timeSinceLastActive > connectionCleanupPeriodMs || - active < poolMinActiveRatio * total) { + active < MIN_ACTIVE_RATIO * total) { // Remove and close 1 connection List conns = pool.removeConnections(1); for (ConnectionContext conn : conns) { @@ -415,9 +412,8 @@ public class ConnectionManager { try { int total = pool.getNumConnections(); int active = pool.getNumActiveConnections(); -float poolMinActiveRatio = pool.getMinActiveRatio(); if (pool.getNumConnections() < pool.getMaxSize() && -active >= poolMinActiveRatio * total) { +active >= MIN_ACTIVE_RATIO * total) { ConnectionContext conn = pool.newConnec
hadoop git commit: HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui.
Repository: hadoop Updated Branches: refs/heads/HDFS-13891 2ddf87f7a -> 7c0d6f65f HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7c0d6f65 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7c0d6f65 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7c0d6f65 Branch: refs/heads/HDFS-13891 Commit: 7c0d6f65fde12ead91ed7c706521ad1d3dc995f8 Parents: 2ddf87f Author: Yiqun Lin Authored: Tue Dec 4 19:58:38 2018 +0800 Committer: Yiqun Lin Committed: Tue Dec 4 19:58:38 2018 +0800 -- .../federation/router/ConnectionManager.java| 20 +--- .../federation/router/ConnectionPool.java | 14 +- .../server/federation/router/RBFConfigKeys.java | 5 ++ .../src/main/resources/hdfs-rbf-default.xml | 8 +++ .../router/TestConnectionManager.java | 51 +--- 5 files changed, 83 insertions(+), 15 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/7c0d6f65/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java index fa2bf94..745 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java @@ -49,10 +49,6 @@ public class ConnectionManager { private static final Logger LOG = LoggerFactory.getLogger(ConnectionManager.class); - /** Minimum amount of active connections: 50%. */ - protected static final float MIN_ACTIVE_RATIO = 0.5f; - - /** Configuration for the connection manager, pool and sockets. */ private final Configuration conf; @@ -60,6 +56,8 @@ public class ConnectionManager { private final int minSize = 1; /** Max number of connections per user + nn. */ private final int maxSize; + /** Min ratio of active connections per user + nn. */ + private final float minActiveRatio; /** How often we close a pool for a particular user + nn. */ private final long poolCleanupPeriodMs; @@ -96,10 +94,13 @@ public class ConnectionManager { public ConnectionManager(Configuration config) { this.conf = config; -// Configure minimum and maximum connection pools +// Configure minimum, maximum and active connection pools this.maxSize = this.conf.getInt( RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_POOL_SIZE, RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_POOL_SIZE_DEFAULT); +this.minActiveRatio = this.conf.getFloat( +RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_MIN_ACTIVE_RATIO, +RBFConfigKeys.DFS_ROUTER_NAMENODE_CONNECTION_MIN_ACTIVE_RATIO_DEFAULT); // Map with the connections indexed by UGI and Namenode this.pools = new HashMap<>(); @@ -203,7 +204,8 @@ public class ConnectionManager { pool = this.pools.get(connectionId); if (pool == null) { pool = new ConnectionPool( - this.conf, nnAddress, ugi, this.minSize, this.maxSize, protocol); + this.conf, nnAddress, ugi, this.minSize, this.maxSize, + this.minActiveRatio, protocol); this.pools.put(connectionId, pool); } } finally { @@ -326,8 +328,9 @@ public class ConnectionManager { long timeSinceLastActive = Time.now() - pool.getLastActiveTime(); int total = pool.getNumConnections(); int active = pool.getNumActiveConnections(); + float poolMinActiveRatio = pool.getMinActiveRatio(); if (timeSinceLastActive > connectionCleanupPeriodMs || - active < MIN_ACTIVE_RATIO * total) { + active < poolMinActiveRatio * total) { // Remove and close 1 connection List conns = pool.removeConnections(1); for (ConnectionContext conn : conns) { @@ -412,8 +415,9 @@ public class ConnectionManager { try { int total = pool.getNumConnections(); int active = pool.getNumActiveConnections(); +float poolMinActiveRatio = pool.getMinActiveRatio(); if (pool.getNumConnections() < pool.getMaxSize() && -active >= MIN_ACTIVE_RATIO * total) { +active >= poolMinActiveRatio * total) { ConnectionContext conn = pool.newConnection(); pool.addConnection(conn); } else {
hadoop git commit: HDDS-848. Create SCM metrics related to container state. Contributed by Bharat Viswanadham.
Repository: hadoop Updated Branches: refs/heads/trunk 042c8ef59 -> 3044b78bd HDDS-848. Create SCM metrics related to container state. Contributed by Bharat Viswanadham. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3044b78b Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3044b78b Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3044b78b Branch: refs/heads/trunk Commit: 3044b78bd0191883d5f9daf2601a58a268beed06 Parents: 042c8ef Author: Yiqun Lin Authored: Mon Dec 3 17:16:34 2018 +0800 Committer: Yiqun Lin Committed: Mon Dec 3 17:16:34 2018 +0800 -- .../hadoop/hdds/scm/server/SCMMXBean.java | 5 ++ .../scm/server/StorageContainerManager.java | 11 +++ .../apache/hadoop/ozone/scm/TestSCMMXBean.java | 81 ++-- 3 files changed, 91 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/3044b78b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java index 4093918..dc09ceb 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java @@ -59,4 +59,9 @@ public interface SCMMXBean extends ServiceRuntimeInfo { * @return String */ double getChillModeCurrentContainerThreshold(); + + /** + * Returns the container count in all states. + */ + Map getContainerStateCount(); } http://git-wip-us.apache.org/repos/asf/hadoop/blob/3044b78b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java index 2d27984..a0d5e1d 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java @@ -30,6 +30,7 @@ import com.google.protobuf.BlockingService; import org.apache.hadoop.classification.InterfaceAudience; import org.apache.hadoop.hdds.HddsUtils; import org.apache.hadoop.hdds.conf.OzoneConfiguration; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.NodeState; import org.apache.hadoop.hdds.scm.ScmConfigKeys; import org.apache.hadoop.hdds.scm.block.BlockManager; @@ -925,6 +926,16 @@ public final class StorageContainerManager extends ServiceRuntimeInfoImpl return scmChillModeManager.getCurrentContainerThreshold(); } + @Override + public Map getContainerStateCount() { +Map nodeStateCount = new HashMap<>(); +for (HddsProtos.LifeCycleState state: HddsProtos.LifeCycleState.values()) { + nodeStateCount.put(state.toString(), containerManager.getContainers( + state).size()); +} +return nodeStateCount; + } + /** * Startup options. */ http://git-wip-us.apache.org/repos/asf/hadoop/blob/3044b78b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestSCMMXBean.java -- diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestSCMMXBean.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestSCMMXBean.java index 3136df2..eabf5e0 100644 --- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestSCMMXBean.java +++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestSCMMXBean.java @@ -20,6 +20,10 @@ package org.apache.hadoop.ozone.scm; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hdds.protocol.proto.HddsProtos; +import org.apache.hadoop.hdds.scm.container.ContainerID; +import org.apache.hadoop.hdds.scm.container.ContainerInfo; +import org.apache.hadoop.hdds.scm.container.ContainerManager; import org.apache.hadoop.hdds.scm.server.StorageContainerManager; import org.apache.hadoop.ozone.MiniOzoneCluster; import org.apache.hadoop.hdds.conf.OzoneConfiguration; @@ -31,8 +35,11 @@ import javax.management.MBeanServer; import javax.management.ObjectName; import java.io.IOException; import java.lang.management.ManagementFactory; +import java.util.ArrayList; +import
hadoop git commit: HDDS-886. Unnecessary buffer copy in HddsDispatcher#dispatch. Contributed by Lokesh Jain.
Repository: hadoop Updated Branches: refs/heads/trunk 7ccb640a6 -> 62f821115 HDDS-886. Unnecessary buffer copy in HddsDispatcher#dispatch. Contributed by Lokesh Jain. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/62f82111 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/62f82111 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/62f82111 Branch: refs/heads/trunk Commit: 62f821115be34f26e994790591c235710f0fc224 Parents: 7ccb640 Author: Yiqun Lin Authored: Fri Nov 30 22:57:29 2018 +0800 Committer: Yiqun Lin Committed: Fri Nov 30 22:57:29 2018 +0800 -- .../apache/hadoop/ozone/container/common/impl/HddsDispatcher.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/62f82111/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java -- diff --git a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java index 24ba784..352cc86 100644 --- a/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java +++ b/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java @@ -134,9 +134,9 @@ public class HddsDispatcher implements ContainerDispatcher, Auditor { @Override public ContainerCommandResponseProto dispatch( ContainerCommandRequestProto msg) { +Preconditions.checkNotNull(msg); LOG.trace("Command {}, trace ID: {} ", msg.getCmdType().toString(), msg.getTraceID()); -Preconditions.checkNotNull(msg.toString()); AuditAction action = ContainerCommandRequestPBHelper.getAuditAction( msg.getCmdType()); - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13870. WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API doc. Contributed by Siyao Meng.
Repository: hadoop Updated Branches: refs/heads/branch-2 408f60e6d -> 50e11dc80 HDFS-13870. WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API doc. Contributed by Siyao Meng. (cherry picked from commit 0e36e935d909862401890d0a5410204504f48b31) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/50e11dc8 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/50e11dc8 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/50e11dc8 Branch: refs/heads/branch-2 Commit: 50e11dc8075e577b9c788b61be19290077cd57c2 Parents: 408f60e Author: Yiqun Lin Authored: Fri Nov 30 11:31:34 2018 +0800 Committer: Yiqun Lin Committed: Fri Nov 30 11:34:28 2018 +0800 -- .../hadoop-hdfs/src/site/markdown/WebHDFS.md| 24 1 file changed, 24 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/50e11dc8/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md index 3d7b32f..20dd8b3 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md @@ -62,6 +62,8 @@ The HTTP REST API supports the complete [FileSystem](../../api/org/apache/hadoop * [`SETTIMES`](#Set_Access_or_Modification_Time) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).setTimes) * [`RENEWDELEGATIONTOKEN`](#Renew_Delegation_Token) (see [DelegationTokenAuthenticator](../../api/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.html).renewDelegationToken) * [`CANCELDELEGATIONTOKEN`](#Cancel_Delegation_Token) (see [DelegationTokenAuthenticator](../../api/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.html).cancelDelegationToken) +* [`ALLOWSNAPSHOT`](#Allow_Snapshot) +* [`DISALLOWSNAPSHOT`](#Disallow_Snapshot) * [`CREATESNAPSHOT`](#Create_Snapshot) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).createSnapshot) * [`RENAMESNAPSHOT`](#Rename_Snapshot) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).renameSnapshot) * [`SETXATTR`](#Set_XAttr) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).setXAttr) @@ -1224,6 +1226,28 @@ See also: [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).listXAttr Snapshot Operations --- +### Allow Snapshot + +* Submit a HTTP PUT request. + +curl -i -X PUT "http://:/webhdfs/v1/?op=ALLOWSNAPSHOT" + +The client receives a response with zero content length on success: + +HTTP/1.1 200 OK +Content-Length: 0 + +### Disallow Snapshot + +* Submit a HTTP PUT request. + +curl -i -X PUT "http://:/webhdfs/v1/?op=DISALLOWSNAPSHOT" + +The client receives a response with zero content length on success: + +HTTP/1.1 200 OK +Content-Length: 0 + ### Create Snapshot * Submit a HTTP PUT request. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13870. WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API doc. Contributed by Siyao Meng.
Repository: hadoop Updated Branches: refs/heads/trunk bad12031f -> 0e36e935d HDFS-13870. WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API doc. Contributed by Siyao Meng. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0e36e935 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0e36e935 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0e36e935 Branch: refs/heads/trunk Commit: 0e36e935d909862401890d0a5410204504f48b31 Parents: bad1203 Author: Yiqun Lin Authored: Fri Nov 30 11:31:34 2018 +0800 Committer: Yiqun Lin Committed: Fri Nov 30 11:31:34 2018 +0800 -- .../hadoop-hdfs/src/site/markdown/WebHDFS.md| 24 1 file changed, 24 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/0e36e935/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md index 383eda0..8661659 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md @@ -64,6 +64,8 @@ The HTTP REST API supports the complete [FileSystem](../../api/org/apache/hadoop * [`SETTIMES`](#Set_Access_or_Modification_Time) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).setTimes) * [`RENEWDELEGATIONTOKEN`](#Renew_Delegation_Token) (see [DelegationTokenAuthenticator](../../api/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.html).renewDelegationToken) * [`CANCELDELEGATIONTOKEN`](#Cancel_Delegation_Token) (see [DelegationTokenAuthenticator](../../api/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.html).cancelDelegationToken) +* [`ALLOWSNAPSHOT`](#Allow_Snapshot) +* [`DISALLOWSNAPSHOT`](#Disallow_Snapshot) * [`CREATESNAPSHOT`](#Create_Snapshot) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).createSnapshot) * [`RENAMESNAPSHOT`](#Rename_Snapshot) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).renameSnapshot) * [`SETXATTR`](#Set_XAttr) (see [FileSystem](../../api/org/apache/hadoop/fs/FileSystem.html).setXAttr) @@ -1302,6 +1304,28 @@ See also: [HDFSErasureCoding](./HDFSErasureCoding.html#Administrative_commands). Snapshot Operations --- +### Allow Snapshot + +* Submit a HTTP PUT request. + +curl -i -X PUT "http://:/webhdfs/v1/?op=ALLOWSNAPSHOT" + +The client receives a response with zero content length on success: + +HTTP/1.1 200 OK +Content-Length: 0 + +### Disallow Snapshot + +* Submit a HTTP PUT request. + +curl -i -X PUT "http://:/webhdfs/v1/?op=DISALLOWSNAPSHOT" + +The client receives a response with zero content length on success: + +HTTP/1.1 200 OK +Content-Length: 0 + ### Create Snapshot * Submit a HTTP PUT request. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-14082. RBF: Add option to fail operations when a subcluster is unavailable. Contributed by Inigo Goiri.
Repository: hadoop Updated Branches: refs/heads/HDFS-13891 4d8cc85c2 -> 175fd4cc4 HDFS-14082. RBF: Add option to fail operations when a subcluster is unavailable. Contributed by Inigo Goiri. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/175fd4cc Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/175fd4cc Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/175fd4cc Branch: refs/heads/HDFS-13891 Commit: 175fd4cc4821a5c197953485c18b72d570435c16 Parents: 4d8cc85 Author: Yiqun Lin Authored: Wed Nov 21 10:40:26 2018 +0800 Committer: Yiqun Lin Committed: Wed Nov 21 10:40:26 2018 +0800 -- .../server/federation/router/RBFConfigKeys.java | 4 ++ .../federation/router/RouterClientProtocol.java | 15 +++-- .../federation/router/RouterRpcServer.java | 9 +++ .../src/main/resources/hdfs-rbf-default.xml | 10 .../router/TestRouterRpcMultiDestination.java | 59 5 files changed, 93 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/175fd4cc/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java index dd72e36..10018fe 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RBFConfigKeys.java @@ -125,6 +125,10 @@ public class RBFConfigKeys extends CommonConfigurationKeysPublic { public static final String DFS_ROUTER_CLIENT_REJECT_OVERLOAD = FEDERATION_ROUTER_PREFIX + "client.reject.overload"; public static final boolean DFS_ROUTER_CLIENT_REJECT_OVERLOAD_DEFAULT = false; + public static final String DFS_ROUTER_ALLOW_PARTIAL_LIST = + FEDERATION_ROUTER_PREFIX + "client.allow-partial-listing"; + public static final boolean DFS_ROUTER_ALLOW_PARTIAL_LIST_DEFAULT = true; + // HDFS Router State Store connection public static final String FEDERATION_FILE_RESOLVER_CLIENT_CLASS = http://git-wip-us.apache.org/repos/asf/hadoop/blob/175fd4cc/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java index de94eaf..2fc5358 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java @@ -111,6 +111,9 @@ public class RouterClientProtocol implements ClientProtocol { private final FileSubclusterResolver subclusterResolver; private final ActiveNamenodeResolver namenodeResolver; + /** If it requires response from all subclusters. */ + private final boolean allowPartialList; + /** Identifier for the super user. */ private final String superUser; /** Identifier for the super group. */ @@ -124,6 +127,10 @@ public class RouterClientProtocol implements ClientProtocol { this.subclusterResolver = rpcServer.getSubclusterResolver(); this.namenodeResolver = rpcServer.getNamenodeResolver(); +this.allowPartialList = conf.getBoolean( +RBFConfigKeys.DFS_ROUTER_ALLOW_PARTIAL_LIST, +RBFConfigKeys.DFS_ROUTER_ALLOW_PARTIAL_LIST_DEFAULT); + // User and group for reporting this.superUser = System.getProperty("user.name"); this.superGroup = conf.get( @@ -606,8 +613,8 @@ public class RouterClientProtocol implements ClientProtocol { new Class[] {String.class, startAfter.getClass(), boolean.class}, new RemoteParam(), startAfter, needLocation); Map listings = -rpcClient.invokeConcurrent( -locations, method, false, false, DirectoryListing.class); +rpcClient.invokeConcurrent(locations, method, +!this.allowPartialList, false, DirectoryListing.class); Map nnListing = new TreeMap<>(); int totalRemainingEntries = 0; @@ -996,8 +1003,8 @@ public class RouterClientProtocol implements ClientProtocol { RemoteMethod method = new RemoteMethod("getCo
svn commit: r1846983 - /hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml
Author: yqlin Date: Tue Nov 20 09:03:30 2018 New Revision: 1846983 URL: http://svn.apache.org/viewvc?rev=1846983&view=rev Log: Add Yiqun to PMC list. Modified: hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml Modified: hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml URL: http://svn.apache.org/viewvc/hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml?rev=1846983&r1=1846982&r2=1846983&view=diff == --- hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml (original) +++ hadoop/common/site/main/author/src/documentation/content/xdocs/who.xml Tue Nov 20 09:03:30 2018 @@ -664,6 +664,14 @@ +yqlin +https://github.com/linyiqun";>Yiqun Lin +Vipshop + ++8 + + + zhz http://zhe-thoughts.github.io/about/";>Zhe Zhang LinkedIn - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDDS-817. Create SCM metrics for disk from node report. Contributed by Bharat Viswanadham.
Repository: hadoop Updated Branches: refs/heads/trunk 1a00b4e32 -> d0cc67944 HDDS-817. Create SCM metrics for disk from node report. Contributed by Bharat Viswanadham. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d0cc6794 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d0cc6794 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d0cc6794 Branch: refs/heads/trunk Commit: d0cc679441da436d7004b38d0eb83af3891e6e09 Parents: 1a00b4e Author: Yiqun Lin Authored: Tue Nov 20 14:22:30 2018 +0800 Committer: Yiqun Lin Committed: Tue Nov 20 14:22:30 2018 +0800 -- .../hadoop/hdds/scm/node/NodeManagerMXBean.java | 7 ++ .../hadoop/hdds/scm/node/NodeStateManager.java | 14 +++ .../hadoop/hdds/scm/node/SCMNodeManager.java| 60 ++ .../hdds/scm/container/MockNodeManager.java | 9 ++ .../testutils/ReplicationNodeManagerMock.java | 5 + .../ozone/scm/TestSCMNodeManagerMXBean.java | 112 +++ 6 files changed, 207 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0cc6794/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManagerMXBean.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManagerMXBean.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManagerMXBean.java index d84cf53..e1b51ef 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManagerMXBean.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManagerMXBean.java @@ -35,4 +35,11 @@ public interface NodeManagerMXBean { * @return A state to number of nodes that in this state mapping */ Map getNodeCount(); + + /** + * Get the disk metrics like capacity, usage and remaining based on the + * storage type. + */ + Map getNodeInfo(); + } http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0cc6794/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStateManager.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStateManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStateManager.java index cddd3ae..a26b0cc 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStateManager.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStateManager.java @@ -274,6 +274,20 @@ public class NodeStateManager implements Runnable, Closeable { } /** + * Get information about the node. + * + * @param datanodeUUID datanode UUID + * + * @return DatanodeInfo + * + * @throws NodeNotFoundException if the node is not present + */ + public DatanodeInfo getNode(UUID datanodeUUID) + throws NodeNotFoundException { +return nodeStateMap.getNodeInfo(datanodeUUID); + } + + /** * Updates the last heartbeat time of the node. * * @throws NodeNotFoundException if the node is not present http://git-wip-us.apache.org/repos/asf/hadoop/blob/d0cc6794/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java index 24672cf..374ff90 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java @@ -19,6 +19,7 @@ package org.apache.hadoop.hdds.scm.node; import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; +import org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos; import org.apache.hadoop.hdds.protocol.proto .StorageContainerDatanodeProtocolProtos.PipelineReportsProto; import org.apache.hadoop.hdds.scm.container.ContainerID; @@ -58,6 +59,7 @@ import org.slf4j.LoggerFactory; import javax.management.ObjectName; import java.io.IOException; import java.net.InetAddress; +import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -182,6 +184,16 @@ public class SCMNodeManager SCMNodeStat stat; try { stat = nodeStateManager.getNodeStat(dnId); + + // Updating the storage report for the datanode. + // I dont think we will get NotFound exception, as we are taking + // nodeInfo from nodeStateMap, as I see it is no
hadoop git commit: HDDS-831. TestOzoneShell in integration-test is flaky. Contributed by Nanda kumar.
Repository: hadoop Updated Branches: refs/heads/trunk 4c465f553 -> f8713f8ad HDDS-831. TestOzoneShell in integration-test is flaky. Contributed by Nanda kumar. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f8713f8a Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f8713f8a Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f8713f8a Branch: refs/heads/trunk Commit: f8713f8adea9d69330933a2cde594ed11ed9520c Parents: 4c465f5 Author: Yiqun Lin Authored: Tue Nov 13 10:38:27 2018 +0800 Committer: Yiqun Lin Committed: Tue Nov 13 10:38:27 2018 +0800 -- .../test/java/org/apache/hadoop/ozone/ozShell/TestOzoneShell.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/f8713f8a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestOzoneShell.java -- diff --git a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestOzoneShell.java b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestOzoneShell.java index 1900024..bd05b92 100644 --- a/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestOzoneShell.java +++ b/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/ozShell/TestOzoneShell.java @@ -266,7 +266,7 @@ public class TestOzoneShell { */ @Test public void testCreateVolumeWithoutUser() throws Exception { -String volumeName = "volume" + RandomStringUtils.randomNumeric(1); +String volumeName = "volume" + RandomStringUtils.randomNumeric(5); String[] args = new String[] {"volume", "create", url + "/" + volumeName, "--root"}; - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDDS-802. Container State Manager should get open pipelines for allocating container. Contributed by Lokesh Jain.
Repository: hadoop Updated Branches: refs/heads/trunk c80f753b0 -> 9317a61f3 HDDS-802. Container State Manager should get open pipelines for allocating container. Contributed by Lokesh Jain. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9317a61f Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9317a61f Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9317a61f Branch: refs/heads/trunk Commit: 9317a61f3cdc5ca91c6934eec9898cee3d65441a Parents: c80f753 Author: Yiqun Lin Authored: Thu Nov 8 23:41:43 2018 +0800 Committer: Yiqun Lin Committed: Thu Nov 8 23:41:43 2018 +0800 -- .../scm/container/ContainerStateManager.java| 4 +- .../hdds/scm/pipeline/PipelineManager.java | 3 + .../hdds/scm/pipeline/PipelineStateManager.java | 5 ++ .../hdds/scm/pipeline/PipelineStateMap.java | 22 +++ .../hdds/scm/pipeline/SCMPipelineManager.java | 11 .../scm/pipeline/TestPipelineStateManager.java | 61 ++-- 6 files changed, 100 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/9317a61f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerStateManager.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerStateManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerStateManager.java index 87505c3..74c8dcb 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerStateManager.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerStateManager.java @@ -248,8 +248,8 @@ public class ContainerStateManager { try { pipeline = pipelineManager.createPipeline(type, replicationFactor); } catch (IOException e) { - final List pipelines = - pipelineManager.getPipelines(type, replicationFactor); + final List pipelines = pipelineManager + .getPipelines(type, replicationFactor, Pipeline.PipelineState.OPEN); if (pipelines.isEmpty()) { throw new IOException("Could not allocate container"); } http://git-wip-us.apache.org/repos/asf/hadoop/blob/9317a61f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManager.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManager.java index 04ec535..cce09f3 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManager.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManager.java @@ -46,6 +46,9 @@ public interface PipelineManager extends Closeable { List getPipelines(ReplicationType type, ReplicationFactor factor); + List getPipelines(ReplicationType type, + ReplicationFactor factor, Pipeline.PipelineState state); + void addContainerToPipeline(PipelineID pipelineID, ContainerID containerID) throws IOException; http://git-wip-us.apache.org/repos/asf/hadoop/blob/9317a61f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateManager.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateManager.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateManager.java index 67f74d3..9f95378 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateManager.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateManager.java @@ -64,6 +64,11 @@ class PipelineStateManager { return pipelineStateMap.getPipelines(type, factor); } + List getPipelines(ReplicationType type, ReplicationFactor factor, + PipelineState state) { +return pipelineStateMap.getPipelines(type, factor, state); + } + List getPipelines(ReplicationType type, PipelineState... states) { return pipelineStateMap.getPipelines(type, states); } http://git-wip-us.apache.org/repos/asf/hadoop/blob/9317a61f/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateMap.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateMap.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateM
hadoop git commit: HDDS-809. Refactor SCMChillModeManager.
Repository: hadoop Updated Branches: refs/heads/trunk 482716e5a -> addec2929 HDDS-809. Refactor SCMChillModeManager. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/addec292 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/addec292 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/addec292 Branch: refs/heads/trunk Commit: addec29297e61a417f0ce711bd76b6db53d504eb Parents: 482716e Author: Yiqun Lin Authored: Wed Nov 7 13:53:28 2018 +0800 Committer: Yiqun Lin Committed: Wed Nov 7 13:54:08 2018 +0800 -- .../org/apache/hadoop/hdds/scm/ScmUtils.java| 2 +- .../hadoop/hdds/scm/block/BlockManagerImpl.java | 2 +- .../hdds/scm/chillmode/ChillModeExitRule.java | 32 ++ .../hdds/scm/chillmode/ChillModePrecheck.java | 68 .../scm/chillmode/ChillModeRestrictedOps.java | 41 +++ .../scm/chillmode/ContainerChillModeRule.java | 112 +++ .../scm/chillmode/DataNodeChillModeRule.java| 83 + .../hadoop/hdds/scm/chillmode/Precheck.java | 29 ++ .../hdds/scm/chillmode/SCMChillModeManager.java | 153 + .../hadoop/hdds/scm/chillmode/package-info.java | 18 ++ .../hdds/scm/server/ChillModePrecheck.java | 69 .../apache/hadoop/hdds/scm/server/Precheck.java | 29 -- .../hdds/scm/server/SCMChillModeManager.java| 319 --- .../scm/server/SCMClientProtocolServer.java | 1 + .../scm/server/StorageContainerManager.java | 1 + .../scm/chillmode/TestSCMChillModeManager.java | 215 + .../scm/server/TestSCMChillModeManager.java | 215 - .../hadoop/ozone/om/TestScmChillMode.java | 2 +- 18 files changed, 756 insertions(+), 635 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/addec292/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ScmUtils.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ScmUtils.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ScmUtils.java index 435f0a5..43b4452 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ScmUtils.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ScmUtils.java @@ -19,8 +19,8 @@ package org.apache.hadoop.hdds.scm; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ScmOps; +import org.apache.hadoop.hdds.scm.chillmode.Precheck; import org.apache.hadoop.hdds.scm.exceptions.SCMException; -import org.apache.hadoop.hdds.scm.server.Precheck; /** * SCM utility class. http://git-wip-us.apache.org/repos/asf/hadoop/blob/addec292/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/BlockManagerImpl.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/BlockManagerImpl.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/BlockManagerImpl.java index c878d97..85658b9 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/BlockManagerImpl.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/BlockManagerImpl.java @@ -23,6 +23,7 @@ import org.apache.hadoop.hdds.client.ContainerBlockID; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ScmOps; import org.apache.hadoop.hdds.scm.ScmConfigKeys; import org.apache.hadoop.hdds.scm.ScmUtils; +import org.apache.hadoop.hdds.scm.chillmode.ChillModePrecheck; import org.apache.hadoop.hdds.scm.container.ContainerManager; import org.apache.hadoop.hdds.scm.container.common.helpers.AllocatedBlock; import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerWithPipeline; @@ -32,7 +33,6 @@ import org.apache.hadoop.hdds.scm.node.NodeManager; import org.apache.hadoop.hdds.protocol.proto.HddsProtos; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationFactor; import org.apache.hadoop.hdds.protocol.proto.HddsProtos.ReplicationType; -import org.apache.hadoop.hdds.scm.server.ChillModePrecheck; import org.apache.hadoop.hdds.server.events.EventHandler; import org.apache.hadoop.hdds.server.events.EventPublisher; import org.apache.hadoop.metrics2.util.MBeans; http://git-wip-us.apache.org/repos/asf/hadoop/blob/addec292/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/ChillModeExitRule.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/ChillModeExitRule.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/chillmode/ChillModeExitRule.java new file mode 100644 index 000..d283dfe --- /dev/null +++ b/hadoop-hdds
hadoop git commit: HDDS-796. Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy.
Repository: hadoop Updated Branches: refs/heads/trunk d43cc5db0 -> 15df2e7a7 HDDS-796. Fix failed test TestStorageContainerManagerHttpServer#testHttpPolicy. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/15df2e7a Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/15df2e7a Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/15df2e7a Branch: refs/heads/trunk Commit: 15df2e7a7547e12e884b624d9f17ad2799d9ccf9 Parents: d43cc5d Author: Yiqun Lin Authored: Mon Nov 5 17:31:06 2018 +0800 Committer: Yiqun Lin Committed: Mon Nov 5 17:31:06 2018 +0800 -- .../java/org/apache/hadoop/hdds/server/BaseHttpServer.java| 7 ++- 1 file changed, 2 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/15df2e7a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/BaseHttpServer.java -- diff --git a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/BaseHttpServer.java b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/BaseHttpServer.java index 2726fc3..5e7d7b8 100644 --- a/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/BaseHttpServer.java +++ b/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/BaseHttpServer.java @@ -115,13 +115,10 @@ public abstract class BaseHttpServer { final Optional addressPort = getPortNumberFromConfigKeys(conf, addressKey); -final Optional addresHost = +final Optional addressHost = getHostNameFromConfigKeys(conf, addressKey); -String hostName = bindHost.orElse(addresHost.get()); -if (hostName == null || hostName.isEmpty()) { - hostName = bindHostDefault; -} +String hostName = bindHost.orElse(addressHost.orElse(bindHostDefault)); return NetUtils.createSocketAddr( hostName + ":" + addressPort.orElse(bindPortdefault)); - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-14049. TestHttpFSServerWebServer fails on Windows because of missing winutils.exe. Contributed by Inigo Goiri.
Repository: hadoop Updated Branches: refs/heads/trunk cb8d679c9 -> 4e3df75eb HDFS-14049. TestHttpFSServerWebServer fails on Windows because of missing winutils.exe. Contributed by Inigo Goiri. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4e3df75e Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4e3df75e Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4e3df75e Branch: refs/heads/trunk Commit: 4e3df75eb72adbab18a1d6476f228a0b504238fa Parents: cb8d679 Author: Yiqun Lin Authored: Sun Nov 4 09:15:53 2018 +0800 Committer: Yiqun Lin Committed: Sun Nov 4 09:15:53 2018 +0800 -- .../hadoop/fs/http/server/TestHttpFSServerWebServer.java | 11 +++ 1 file changed, 11 insertions(+) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/4e3df75e/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerWebServer.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerWebServer.java b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerWebServer.java index 5250543..97d41d3 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerWebServer.java +++ b/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServerWebServer.java @@ -30,6 +30,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.security.authentication.server.AuthenticationFilter; import org.apache.hadoop.test.GenericTestUtils; import org.apache.hadoop.test.HadoopUsersConfTestHelper; +import org.apache.hadoop.util.Shell; import org.junit.Assert; import org.junit.Before; import org.junit.BeforeClass; @@ -55,6 +56,16 @@ public class TestHttpFSServerWebServer { confDir.mkdirs(); logsDir.mkdirs(); tempDir.mkdirs(); + +if (Shell.WINDOWS) { + File binDir = new File(homeDir, "bin"); + binDir.mkdirs(); + File winutils = Shell.getWinUtilsFile(); + if (winutils.exists()) { +FileUtils.copyFileToDirectory(winutils, binDir); + } +} + System.setProperty("hadoop.home.dir", homeDir.getAbsolutePath()); System.setProperty("hadoop.log.dir", logsDir.getAbsolutePath()); System.setProperty("httpfs.home.dir", homeDir.getAbsolutePath()); - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDDS-751. Replace usage of Guava Optional with Java Optional.
Repository: hadoop Updated Branches: refs/heads/trunk 8fe85af63 -> d16d5f730 HDDS-751. Replace usage of Guava Optional with Java Optional. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d16d5f73 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d16d5f73 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d16d5f73 Branch: refs/heads/trunk Commit: d16d5f730e9d139d3e026805f21ac2c9b0bbb98b Parents: 8fe85af Author: Yiqun Lin Authored: Fri Nov 2 10:50:32 2018 +0800 Committer: Yiqun Lin Committed: Fri Nov 2 10:50:32 2018 +0800 -- .../java/org/apache/hadoop/hdds/HddsUtils.java | 29 +--- .../apache/hadoop/hdds/scm/HddsServerUtil.java | 17 ++-- .../hadoop/hdds/server/BaseHttpServer.java | 9 -- .../java/org/apache/hadoop/ozone/OmUtils.java | 8 +++--- 4 files changed, 32 insertions(+), 31 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/d16d5f73/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java -- diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java index 89edfdd..18637af 100644 --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java @@ -18,7 +18,6 @@ package org.apache.hadoop.hdds; -import com.google.common.base.Optional; import com.google.common.base.Strings; import com.google.common.net.HostAndPort; import org.apache.hadoop.classification.InterfaceAudience; @@ -45,6 +44,7 @@ import java.util.Calendar; import java.util.Collection; import java.util.HashSet; import java.util.Map; +import java.util.Optional; import java.util.TimeZone; import static org.apache.hadoop.hdfs.DFSConfigKeys @@ -114,7 +114,7 @@ public final class HddsUtils { ScmConfigKeys.OZONE_SCM_CLIENT_ADDRESS_KEY); return NetUtils.createSocketAddr(host.get() + ":" + port -.or(ScmConfigKeys.OZONE_SCM_CLIENT_PORT_DEFAULT)); +.orElse(ScmConfigKeys.OZONE_SCM_CLIENT_PORT_DEFAULT)); } /** @@ -162,7 +162,7 @@ public final class HddsUtils { ScmConfigKeys.OZONE_SCM_BLOCK_CLIENT_ADDRESS_KEY); return NetUtils.createSocketAddr(host.get() + ":" + port -.or(ScmConfigKeys.OZONE_SCM_BLOCK_CLIENT_PORT_DEFAULT)); +.orElse(ScmConfigKeys.OZONE_SCM_BLOCK_CLIENT_PORT_DEFAULT)); } /** @@ -186,7 +186,7 @@ public final class HddsUtils { return hostName; } } -return Optional.absent(); +return Optional.empty(); } /** @@ -196,7 +196,7 @@ public final class HddsUtils { */ public static Optional getHostName(String value) { if ((value == null) || value.isEmpty()) { - return Optional.absent(); + return Optional.empty(); } return Optional.of(HostAndPort.fromString(value).getHostText()); } @@ -208,11 +208,11 @@ public final class HddsUtils { */ public static Optional getHostPort(String value) { if ((value == null) || value.isEmpty()) { - return Optional.absent(); + return Optional.empty(); } int port = HostAndPort.fromString(value).getPortOrDefault(NO_PORT); if (port == NO_PORT) { - return Optional.absent(); + return Optional.empty(); } else { return Optional.of(port); } @@ -239,7 +239,7 @@ public final class HddsUtils { return hostPort; } } -return Optional.absent(); +return Optional.empty(); } /** @@ -261,20 +261,17 @@ public final class HddsUtils { + " Null or empty address list found."); } -final com.google.common.base.Optional -defaultPort = com.google.common.base.Optional.of(ScmConfigKeys -.OZONE_SCM_DEFAULT_PORT); +final Optional defaultPort = Optional +.of(ScmConfigKeys.OZONE_SCM_DEFAULT_PORT); for (String address : names) { - com.google.common.base.Optional hostname = - getHostName(address); + Optional hostname = getHostName(address); if (!hostname.isPresent()) { throw new IllegalArgumentException("Invalid hostname for SCM: " + hostname); } - com.google.common.base.Optional port = - getHostPort(address); + Optional port = getHostPort(address); InetSocketAddress addr = NetUtils.createSocketAddr(hostname.get(), - port.or(defaultPort.get())); + port.orElse(defaultPort.get())); addresses.add(addr); } return addresses; http://git-wip-us.apache.org/repos/asf/hadoop/blob/d16d5f73/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/scm/HddsServerUtil.java ---
hadoop git commit: HDDS-786. Fix the findbugs for SCMClientProtocolServer#getContainerWithPipeline.
Repository: hadoop Updated Branches: refs/heads/ozone-0.3 8411c2bf5 -> 73e9e4348 HDDS-786. Fix the findbugs for SCMClientProtocolServer#getContainerWithPipeline. (cherry picked from commit 2e8ac14dcb57a0fe07b2119c26535c3541665b70) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/73e9e434 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/73e9e434 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/73e9e434 Branch: refs/heads/ozone-0.3 Commit: 73e9e43483da50707fa22c070b0a8deba29eb8b2 Parents: 8411c2b Author: Yiqun Lin Authored: Thu Nov 1 14:10:17 2018 +0800 Committer: Yiqun Lin Committed: Thu Nov 1 14:12:30 2018 +0800 -- .../org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java | 1 - 1 file changed, 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/73e9e434/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java index 3d228fa..db5a9eb 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java @@ -188,7 +188,6 @@ public class SCMClientProtocolServer implements } } } -String remoteUser = getRpcRemoteUsername(); getScm().checkAdminAccess(null); return scm.getContainerManager() .getContainerWithPipeline(containerID); - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDDS-786. Fix the findbugs for SCMClientProtocolServer#getContainerWithPipeline.
Repository: hadoop Updated Branches: refs/heads/trunk b13c56742 -> 2e8ac14dc HDDS-786. Fix the findbugs for SCMClientProtocolServer#getContainerWithPipeline. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2e8ac14d Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2e8ac14d Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2e8ac14d Branch: refs/heads/trunk Commit: 2e8ac14dcb57a0fe07b2119c26535c3541665b70 Parents: b13c567 Author: Yiqun Lin Authored: Thu Nov 1 14:10:17 2018 +0800 Committer: Yiqun Lin Committed: Thu Nov 1 14:10:17 2018 +0800 -- .../org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java | 1 - 1 file changed, 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/2e8ac14d/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java -- diff --git a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java index e92200a..58cb871 100644 --- a/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java +++ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java @@ -189,7 +189,6 @@ public class SCMClientProtocolServer implements } } } -String remoteUser = getRpcRemoteUsername(); getScm().checkAdminAccess(null); return scm.getContainerManager() .getContainerWithPipeline(ContainerID.valueof(containerID)); - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-14011. RBF: Add more information to HdfsFileStatus for a mount point. Contributed by Akira Ajisaka.
Repository: hadoop Updated Branches: refs/heads/HDFS-13891 ebf6bf304 -> f1566ca85 HDFS-14011. RBF: Add more information to HdfsFileStatus for a mount point. Contributed by Akira Ajisaka. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f1566ca8 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f1566ca8 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f1566ca8 Branch: refs/heads/HDFS-13891 Commit: f1566ca85afdc39d3e62d98a1d06a0a07f0055d5 Parents: ebf6bf3 Author: Yiqun Lin Authored: Tue Oct 23 14:34:29 2018 +0800 Committer: Yiqun Lin Committed: Tue Oct 23 14:34:29 2018 +0800 -- .../resolver/FileSubclusterResolver.java| 6 ++- .../federation/router/RouterClientProtocol.java | 30 --- .../router/RouterQuotaUpdateService.java| 9 ++-- .../hdfs/server/federation/MockResolver.java| 17 +++--- .../federation/router/TestRouterMountTable.java | 55 +++- .../router/TestRouterRpcMultiDestination.java | 5 +- 6 files changed, 97 insertions(+), 25 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/f1566ca8/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/FileSubclusterResolver.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/FileSubclusterResolver.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/FileSubclusterResolver.java index 5aa5ec9..6432bb0 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/FileSubclusterResolver.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/FileSubclusterResolver.java @@ -61,8 +61,10 @@ public interface FileSubclusterResolver { * cache. * * @param path Path to get the mount points under. - * @return List of mount points present at this path or zero-length list if - * none are found. + * @return List of mount points present at this path. Return zero-length + * list if the path is a mount point but there are no mount points + * under the path. Return null if the path is not a mount point + * and there are no mount points under the path. * @throws IOException Throws exception if the data is not available. */ List getMountPoints(String path) throws IOException; http://git-wip-us.apache.org/repos/asf/hadoop/blob/f1566ca8/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java index ddbc014..de94eaf 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java @@ -718,6 +718,9 @@ public class RouterClientProtocol implements ClientProtocol { date = dates.get(src); } ret = getMountPointStatus(src, children.size(), date); + } else if (children != null) { +// The src is a mount point, but there are no files or directories +ret = getMountPointStatus(src, 0, 0); } } @@ -1714,13 +1717,26 @@ public class RouterClientProtocol implements ClientProtocol { FsPermission permission = FsPermission.getDirDefault(); String owner = this.superUser; String group = this.superGroup; -try { - // TODO support users, it should be the user for the pointed folder - UserGroupInformation ugi = RouterRpcServer.getRemoteUser(); - owner = ugi.getUserName(); - group = ugi.getPrimaryGroupName(); -} catch (IOException e) { - LOG.error("Cannot get the remote user: {}", e.getMessage()); +if (subclusterResolver instanceof MountTableResolver) { + try { +MountTableResolver mountTable = (MountTableResolver) subclusterResolver; +MountTable entry = mountTable.getMountPoint(name); +if (entry != null) { + permission = entry.getMode(); + owner = entry.getOwnerName(); + group = entry.getGroupName(); +} + } catch (IOException e) { +LOG.error("Cannot get mount point: {}", e.getMessage()); + } +} else { + try
hadoop git commit: HDDS-628. Fix outdated names used in HDDS documentations.
Repository: hadoop Updated Branches: refs/heads/ozone-0.3 67d516eb8 -> 54a229c33 HDDS-628. Fix outdated names used in HDDS documentations. (cherry picked from commit 5da042227cfce440eddc263b377a70ed5b3743fb) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/54a229c3 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/54a229c3 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/54a229c3 Branch: refs/heads/ozone-0.3 Commit: 54a229c332fb24e3232c933dc675f96888964f3e Parents: 67d516e Author: Yiqun Lin Authored: Fri Oct 12 14:00:13 2018 +0800 Committer: Yiqun Lin Committed: Fri Oct 12 14:01:47 2018 +0800 -- hadoop-ozone/docs/content/Dozone.md | 4 ++-- hadoop-ozone/docs/content/JavaApi.md | 2 +- hadoop-ozone/docs/content/OzoneManager.md | 2 +- hadoop-ozone/docs/content/Settings.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/54a229c3/hadoop-ozone/docs/content/Dozone.md -- diff --git a/hadoop-ozone/docs/content/Dozone.md b/hadoop-ozone/docs/content/Dozone.md index 7906cf3..f6efb0f 100644 --- a/hadoop-ozone/docs/content/Dozone.md +++ b/hadoop-ozone/docs/content/Dozone.md @@ -63,14 +63,14 @@ Useful Docker & Ozone Commands If you make any modifications to ozone, the simplest way to test it is to run freon and unit tests. -Here are the instructions to run corona in a docker based cluster. +Here are the instructions to run freon in a docker based cluster. {{< highlight bash >}} docker-compose exec datanode bash {{< /highlight >}} This will open a bash shell on the data node container. -Now we can execute corona for load generation. +Now we can execute freon for load generation. {{< highlight bash >}} ozone freon randomkeys --numOfVolumes=10 --numOfBuckets 10 --numOfKeys 10 http://git-wip-us.apache.org/repos/asf/hadoop/blob/54a229c3/hadoop-ozone/docs/content/JavaApi.md -- diff --git a/hadoop-ozone/docs/content/JavaApi.md b/hadoop-ozone/docs/content/JavaApi.md index 1d32bed..e538f4b 100644 --- a/hadoop-ozone/docs/content/JavaApi.md +++ b/hadoop-ozone/docs/content/JavaApi.md @@ -42,7 +42,7 @@ can use OzoneClient ozClient = OzoneClientFactory.getRestClient(); {{< /highlight >}} -And to get a a RPC client we can call +And to get a RPC client we can call {{< highlight java >}} OzoneClient ozClient = OzoneClientFactory.getRpcClient(); http://git-wip-us.apache.org/repos/asf/hadoop/blob/54a229c3/hadoop-ozone/docs/content/OzoneManager.md -- diff --git a/hadoop-ozone/docs/content/OzoneManager.md b/hadoop-ozone/docs/content/OzoneManager.md index 560f827..5eb8663 100644 --- a/hadoop-ozone/docs/content/OzoneManager.md +++ b/hadoop-ozone/docs/content/OzoneManager.md @@ -70,7 +70,7 @@ We are hopeful that this leads to a more straightforward way of discovering sett OM and SCM --- [Storage container manager]({{< ref "Hdds.md" >}}) or (SCM) is the block manager - for ozone. When a client requests OM for a set of data nodes to write data, OM talk to SCM and gets a block. + for ozone. When a client requests OM for a set of data nodes to write data, OM talks to SCM and gets a block. A block returned by SCM contains a pipeline, which is a set of nodes that we participate in that block replication. http://git-wip-us.apache.org/repos/asf/hadoop/blob/54a229c3/hadoop-ozone/docs/content/Settings.md -- diff --git a/hadoop-ozone/docs/content/Settings.md b/hadoop-ozone/docs/content/Settings.md index 41ab04a..5c9bb41 100644 --- a/hadoop-ozone/docs/content/Settings.md +++ b/hadoop-ozone/docs/content/Settings.md @@ -43,7 +43,7 @@ requests blocks from SCM, to which clients can write data. ## Setting up an Ozone only cluster -* Please untar the ozone-0.2.1-SNAPSHOT to the directory where you are going +* Please untar the ozone- to the directory where you are going to run Ozone from. We need Ozone jars on all machines in the cluster. So you need to do this on all machines in the cluster. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDDS-628. Fix outdated names used in HDDS documentations.
Repository: hadoop Updated Branches: refs/heads/trunk bca928d3c -> 5da042227 HDDS-628. Fix outdated names used in HDDS documentations. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/5da04222 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/5da04222 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/5da04222 Branch: refs/heads/trunk Commit: 5da042227cfce440eddc263b377a70ed5b3743fb Parents: bca928d Author: Yiqun Lin Authored: Fri Oct 12 14:00:13 2018 +0800 Committer: Yiqun Lin Committed: Fri Oct 12 14:00:13 2018 +0800 -- hadoop-ozone/docs/content/Dozone.md | 4 ++-- hadoop-ozone/docs/content/JavaApi.md | 2 +- hadoop-ozone/docs/content/OzoneManager.md | 2 +- hadoop-ozone/docs/content/Settings.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/5da04222/hadoop-ozone/docs/content/Dozone.md -- diff --git a/hadoop-ozone/docs/content/Dozone.md b/hadoop-ozone/docs/content/Dozone.md index 7906cf3..f6efb0f 100644 --- a/hadoop-ozone/docs/content/Dozone.md +++ b/hadoop-ozone/docs/content/Dozone.md @@ -63,14 +63,14 @@ Useful Docker & Ozone Commands If you make any modifications to ozone, the simplest way to test it is to run freon and unit tests. -Here are the instructions to run corona in a docker based cluster. +Here are the instructions to run freon in a docker based cluster. {{< highlight bash >}} docker-compose exec datanode bash {{< /highlight >}} This will open a bash shell on the data node container. -Now we can execute corona for load generation. +Now we can execute freon for load generation. {{< highlight bash >}} ozone freon randomkeys --numOfVolumes=10 --numOfBuckets 10 --numOfKeys 10 http://git-wip-us.apache.org/repos/asf/hadoop/blob/5da04222/hadoop-ozone/docs/content/JavaApi.md -- diff --git a/hadoop-ozone/docs/content/JavaApi.md b/hadoop-ozone/docs/content/JavaApi.md index 1d32bed..e538f4b 100644 --- a/hadoop-ozone/docs/content/JavaApi.md +++ b/hadoop-ozone/docs/content/JavaApi.md @@ -42,7 +42,7 @@ can use OzoneClient ozClient = OzoneClientFactory.getRestClient(); {{< /highlight >}} -And to get a a RPC client we can call +And to get a RPC client we can call {{< highlight java >}} OzoneClient ozClient = OzoneClientFactory.getRpcClient(); http://git-wip-us.apache.org/repos/asf/hadoop/blob/5da04222/hadoop-ozone/docs/content/OzoneManager.md -- diff --git a/hadoop-ozone/docs/content/OzoneManager.md b/hadoop-ozone/docs/content/OzoneManager.md index 560f827..5eb8663 100644 --- a/hadoop-ozone/docs/content/OzoneManager.md +++ b/hadoop-ozone/docs/content/OzoneManager.md @@ -70,7 +70,7 @@ We are hopeful that this leads to a more straightforward way of discovering sett OM and SCM --- [Storage container manager]({{< ref "Hdds.md" >}}) or (SCM) is the block manager - for ozone. When a client requests OM for a set of data nodes to write data, OM talk to SCM and gets a block. + for ozone. When a client requests OM for a set of data nodes to write data, OM talks to SCM and gets a block. A block returned by SCM contains a pipeline, which is a set of nodes that we participate in that block replication. http://git-wip-us.apache.org/repos/asf/hadoop/blob/5da04222/hadoop-ozone/docs/content/Settings.md -- diff --git a/hadoop-ozone/docs/content/Settings.md b/hadoop-ozone/docs/content/Settings.md index 41ab04a..5c9bb41 100644 --- a/hadoop-ozone/docs/content/Settings.md +++ b/hadoop-ozone/docs/content/Settings.md @@ -43,7 +43,7 @@ requests blocks from SCM, to which clients can write data. ## Setting up an Ozone only cluster -* Please untar the ozone-0.2.1-SNAPSHOT to the directory where you are going +* Please untar the ozone- to the directory where you are going to run Ozone from. We need Ozone jars on all machines in the cluster. So you need to do this on all machines in the cluster. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13967. HDFS Router Quota Class Review. Contributed by BELUGA BEHR.
Repository: hadoop Updated Branches: refs/heads/trunk 9bbeb5248 -> d4626b4d1 HDFS-13967. HDFS Router Quota Class Review. Contributed by BELUGA BEHR. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d4626b4d Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d4626b4d Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d4626b4d Branch: refs/heads/trunk Commit: d4626b4d1825b60ef02c0da9c45cd483d1d98f49 Parents: 9bbeb52 Author: Yiqun Lin Authored: Tue Oct 9 16:11:07 2018 +0800 Committer: Yiqun Lin Committed: Tue Oct 9 16:11:07 2018 +0800 -- .../hdfs/server/federation/router/Quota.java| 54 ++-- 1 file changed, 26 insertions(+), 28 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/d4626b4d/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java index d8ed080..5d0309f 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Quota.java @@ -18,12 +18,14 @@ package org.apache.hadoop.hdfs.server.federation.router; import java.io.IOException; -import java.util.HashMap; -import java.util.LinkedList; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; import java.util.List; import java.util.Map; import java.util.Set; +import org.apache.commons.lang3.StringUtils; import org.apache.hadoop.fs.QuotaUsage; import org.apache.hadoop.fs.StorageType; import org.apache.hadoop.hdfs.protocol.ClientProtocol; @@ -33,6 +35,9 @@ import org.apache.hadoop.hdfs.server.namenode.NameNode.OperationCategory; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import com.google.common.collect.ArrayListMultimap; +import com.google.common.collect.ListMultimap; + /** * Module that implements the quota relevant RPC calls * {@link ClientProtocol#setQuota(String, long, long, StorageType)} @@ -121,37 +126,31 @@ public class Quota { final List locations = getQuotaRemoteLocations(path); // NameService -> Locations -Map> validLocations = new HashMap<>(); +ListMultimap validLocations = +ArrayListMultimap.create(); + for (RemoteLocation loc : locations) { - String nsId = loc.getNameserviceId(); - List dests = validLocations.get(nsId); - if (dests == null) { -dests = new LinkedList<>(); -dests.add(loc); -validLocations.put(nsId, dests); - } else { -// Ensure the paths in the same nameservice is different. -// Don't include parent-child paths. -boolean isChildPath = false; -for (RemoteLocation d : dests) { - if (loc.getDest().startsWith(d.getDest())) { -isChildPath = true; -break; - } -} + final String nsId = loc.getNameserviceId(); + final Collection dests = validLocations.get(nsId); + + // Ensure the paths in the same nameservice is different. + // Do not include parent-child paths. + boolean isChildPath = false; -if (!isChildPath) { - dests.add(loc); + for (RemoteLocation d : dests) { +if (StringUtils.startsWith(loc.getDest(), d.getDest())) { + isChildPath = true; + break; } } -} -List quotaLocs = new LinkedList<>(); -for (List locs : validLocations.values()) { - quotaLocs.addAll(locs); + if (!isChildPath) { +validLocations.put(nsId, loc); + } } -return quotaLocs; +return Collections +.unmodifiableList(new ArrayList<>(validLocations.values())); } /** @@ -209,7 +208,7 @@ public class Quota { */ private List getQuotaRemoteLocations(String path) throws IOException { -List locations = new LinkedList<>(); +List locations = new ArrayList<>(); RouterQuotaManager manager = this.router.getQuotaManager(); if (manager != null) { Set childrenPaths = manager.getPaths(path); @@ -217,7 +216,6 @@ public class Quota { locations.addAll(rpcServer.getLocationsForPath(childPath, true, false)); } } - return locations; } } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13962. Add null check for add-replica pool to avoid lock acquiring. Contributed by Surendra Singh Lilhore.
Repository: hadoop Updated Branches: refs/heads/branch-3.1 dd445e036 -> 665036c5f HDFS-13962. Add null check for add-replica pool to avoid lock acquiring. Contributed by Surendra Singh Lilhore. (cherry picked from commit 1043795f7fe44c98a34f8ea3cea708c801c3043b) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/665036c5 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/665036c5 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/665036c5 Branch: refs/heads/branch-3.1 Commit: 665036c5f71f6ce7ea331706ae1deb56da0fd0eb Parents: dd445e0 Author: Yiqun Lin Authored: Tue Oct 9 10:33:13 2018 +0800 Committer: Yiqun Lin Committed: Tue Oct 9 10:36:45 2018 +0800 -- .../hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java| 6 -- .../hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/665036c5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java index d7e56fd..e725834 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java @@ -183,8 +183,10 @@ class BlockPoolSlice { .setConf(conf) .setInitialUsed(loadDfsUsed()) .build(); -// initialize add replica fork join pool -initializeAddReplicaPool(conf); +if (addReplicaThreadPool == null) { + // initialize add replica fork join pool + initializeAddReplicaPool(conf); +} // Make the dfs usage to be saved during shutdown. shutdownHook = new Runnable() { @Override http://git-wip-us.apache.org/repos/asf/hadoop/blob/665036c5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java index c630b95..73d3c60 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java @@ -154,7 +154,7 @@ class ReplicaMap { if (oldReplicaInfo != null) { return oldReplicaInfo; } else { -set.add(replicaInfo); +set.addOrReplace(replicaInfo); } return replicaInfo; } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13962. Add null check for add-replica pool to avoid lock acquiring. Contributed by Surendra Singh Lilhore.
Repository: hadoop Updated Branches: refs/heads/branch-3.2 21553e22b -> b6698e2a8 HDFS-13962. Add null check for add-replica pool to avoid lock acquiring. Contributed by Surendra Singh Lilhore. (cherry picked from commit 1043795f7fe44c98a34f8ea3cea708c801c3043b) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b6698e2a Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b6698e2a Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b6698e2a Branch: refs/heads/branch-3.2 Commit: b6698e2a828a652e995d4cfe83d8fcd095fdeee2 Parents: 21553e2 Author: Yiqun Lin Authored: Tue Oct 9 10:33:13 2018 +0800 Committer: Yiqun Lin Committed: Tue Oct 9 10:35:08 2018 +0800 -- .../hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java| 6 -- .../hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6698e2a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java index b9b581f..4a4fef9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java @@ -183,8 +183,10 @@ class BlockPoolSlice { .setConf(conf) .setInitialUsed(loadDfsUsed()) .build(); -// initialize add replica fork join pool -initializeAddReplicaPool(conf); +if (addReplicaThreadPool == null) { + // initialize add replica fork join pool + initializeAddReplicaPool(conf); +} // Make the dfs usage to be saved during shutdown. shutdownHook = new Runnable() { @Override http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6698e2a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java index c630b95..73d3c60 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java @@ -154,7 +154,7 @@ class ReplicaMap { if (oldReplicaInfo != null) { return oldReplicaInfo; } else { -set.add(replicaInfo); +set.addOrReplace(replicaInfo); } return replicaInfo; } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13962. Add null check for add-replica pool to avoid lock acquiring. Contributed by Surendra Singh Lilhore.
Repository: hadoop Updated Branches: refs/heads/trunk 347ea3858 -> 1043795f7 HDFS-13962. Add null check for add-replica pool to avoid lock acquiring. Contributed by Surendra Singh Lilhore. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1043795f Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1043795f Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1043795f Branch: refs/heads/trunk Commit: 1043795f7fe44c98a34f8ea3cea708c801c3043b Parents: 347ea38 Author: Yiqun Lin Authored: Tue Oct 9 10:33:13 2018 +0800 Committer: Yiqun Lin Committed: Tue Oct 9 10:33:13 2018 +0800 -- .../hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java| 6 -- .../hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/1043795f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java index b9b581f..4a4fef9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java @@ -183,8 +183,10 @@ class BlockPoolSlice { .setConf(conf) .setInitialUsed(loadDfsUsed()) .build(); -// initialize add replica fork join pool -initializeAddReplicaPool(conf); +if (addReplicaThreadPool == null) { + // initialize add replica fork join pool + initializeAddReplicaPool(conf); +} // Make the dfs usage to be saved during shutdown. shutdownHook = new Runnable() { @Override http://git-wip-us.apache.org/repos/asf/hadoop/blob/1043795f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java index c630b95..73d3c60 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java @@ -154,7 +154,7 @@ class ReplicaMap { if (oldReplicaInfo != null) { return oldReplicaInfo; } else { -set.add(replicaInfo); +set.addOrReplace(replicaInfo); } return replicaInfo; } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore.
Repository: hadoop Updated Branches: refs/heads/branch-2 5539dd97d -> c632c6e6e HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c632c6e6 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c632c6e6 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c632c6e6 Branch: refs/heads/branch-2 Commit: c632c6e6e99e3f3722774c9fc269ace88aa5d9bb Parents: 5539dd9 Author: Yiqun Lin Authored: Tue Oct 9 10:20:37 2018 +0800 Committer: Yiqun Lin Committed: Tue Oct 9 10:20:37 2018 +0800 -- .../org/apache/hadoop/hdfs/DFSConfigKeys.java | 3 + .../hadoop/hdfs/server/datanode/DataNode.java | 2 +- .../datanode/fsdataset/impl/BlockPoolSlice.java | 182 +-- .../datanode/fsdataset/impl/FsDatasetImpl.java | 14 ++ .../datanode/fsdataset/impl/FsDatasetUtil.java | 30 +-- .../datanode/fsdataset/impl/FsVolumeList.java | 5 +- .../datanode/fsdataset/impl/ReplicaMap.java | 24 +++ .../src/main/resources/hdfs-default.xml | 9 + .../fsdataset/impl/FsDatasetImplTestUtils.java | 14 +- .../fsdataset/impl/TestFsVolumeList.java| 62 +++ 10 files changed, 315 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/c632c6e6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java index a72ff87..edd99f2 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java @@ -304,6 +304,9 @@ public class DFSConfigKeys extends CommonConfigurationKeys { public static final longDFS_CONTENT_SUMMARY_SLEEP_MICROSEC_DEFAULT = 500; public static final String DFS_DATANODE_FAILED_VOLUMES_TOLERATED_KEY = "dfs.datanode.failed.volumes.tolerated"; public static final int DFS_DATANODE_FAILED_VOLUMES_TOLERATED_DEFAULT = 0; + public static final String + DFS_DATANODE_VOLUMES_REPLICA_ADD_THREADPOOL_SIZE_KEY = + "dfs.datanode.volumes.replica-add.threadpool.size"; public static final String DFS_DATANODE_SYNCONCLOSE_KEY = "dfs.datanode.synconclose"; public static final boolean DFS_DATANODE_SYNCONCLOSE_DEFAULT = false; public static final String DFS_DATANODE_SOCKET_REUSE_KEEPALIVE_KEY = "dfs.datanode.socket.reuse.keepalive"; http://git-wip-us.apache.org/repos/asf/hadoop/blob/c632c6e6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java index dfca5d6..80a7ca2 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java @@ -1630,7 +1630,7 @@ public class DataNode extends ReconfigurableBase return blockPoolManager.get(bpid); } - int getBpOsCount() { + public int getBpOsCount() { return blockPoolManager.getAllNamenodeThreads().size(); } http://git-wip-us.apache.org/repos/asf/hadoop/blob/c632c6e6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java index f46b6a4..0ed5c39 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java @@ -28,8 +28,19 @@ import java.io.InputStream; import java.io.OutputStreamWriter; import java.io.RandomAccessFile; import java.io.Writer; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.Comparator; import java.util.Iterator; +import java.util.List; +import java.util.Queue; import java.util.Scanner; +import java.u
hadoop git commit: HDFS-13957. Fix incorrect option used in description of InMemoryAliasMap.
Repository: hadoop Updated Branches: refs/heads/branch-3.2 b3ac88693 -> 62d02eecd HDFS-13957. Fix incorrect option used in description of InMemoryAliasMap. (cherry picked from commit 619e490333fa89601fd476dedac6d16610e9a52a) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/62d02eec Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/62d02eec Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/62d02eec Branch: refs/heads/branch-3.2 Commit: 62d02eecd0079a9f1fbfb18743c5324a61a03a7c Parents: b3ac886 Author: Yiqun Lin Authored: Fri Oct 5 09:55:08 2018 +0800 Committer: Yiqun Lin Committed: Fri Oct 5 10:09:22 2018 +0800 -- .../hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/62d02eec/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md index b8d5321..21145e6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md @@ -152,7 +152,7 @@ Currently, the following two types of alias maps are supported. This is a LevelDB-based alias map that runs as a separate server in Namenode. The alias map itself can be created using the `fs2img` tool using the option -`-Ddfs.provided.aliasmap.leveldb.path=file:///path/to/leveldb/map/dingos.db -o org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.LevelDBFileRegionAliasMap` +`-Ddfs.provided.aliasmap.leveldb.path=file:///path/to/leveldb/map/dingos.db -b org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.LevelDBFileRegionAliasMap` as in the example above. Datanodes contact this alias map using the `org.apache.hadoop.hdfs.server.aliasmap.InMemoryAliasMapProtocol` protocol. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13957. Fix incorrect option used in description of InMemoryAliasMap.
Repository: hadoop Updated Branches: refs/heads/branch-3.1 86a1ad442 -> dd70adf31 HDFS-13957. Fix incorrect option used in description of InMemoryAliasMap. (cherry picked from commit 619e490333fa89601fd476dedac6d16610e9a52a) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/dd70adf3 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/dd70adf3 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/dd70adf3 Branch: refs/heads/branch-3.1 Commit: dd70adf3184ce6a280df51bd84cfcfdadbb33b32 Parents: 86a1ad4 Author: Yiqun Lin Authored: Fri Oct 5 09:55:08 2018 +0800 Committer: Yiqun Lin Committed: Fri Oct 5 10:08:28 2018 +0800 -- .../hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/dd70adf3/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md index b8d5321..21145e6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md @@ -152,7 +152,7 @@ Currently, the following two types of alias maps are supported. This is a LevelDB-based alias map that runs as a separate server in Namenode. The alias map itself can be created using the `fs2img` tool using the option -`-Ddfs.provided.aliasmap.leveldb.path=file:///path/to/leveldb/map/dingos.db -o org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.LevelDBFileRegionAliasMap` +`-Ddfs.provided.aliasmap.leveldb.path=file:///path/to/leveldb/map/dingos.db -b org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.LevelDBFileRegionAliasMap` as in the example above. Datanodes contact this alias map using the `org.apache.hadoop.hdfs.server.aliasmap.InMemoryAliasMapProtocol` protocol. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13957. Fix incorrect option used in description of InMemoryAliasMap.
Repository: hadoop Updated Branches: refs/heads/trunk cc2babc1f -> 619e49033 HDFS-13957. Fix incorrect option used in description of InMemoryAliasMap. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/619e4903 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/619e4903 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/619e4903 Branch: refs/heads/trunk Commit: 619e490333fa89601fd476dedac6d16610e9a52a Parents: cc2babc Author: Yiqun Lin Authored: Fri Oct 5 09:55:08 2018 +0800 Committer: Yiqun Lin Committed: Fri Oct 5 09:55:08 2018 +0800 -- .../hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/619e4903/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md index b8d5321..21145e6 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsProvidedStorage.md @@ -152,7 +152,7 @@ Currently, the following two types of alias maps are supported. This is a LevelDB-based alias map that runs as a separate server in Namenode. The alias map itself can be created using the `fs2img` tool using the option -`-Ddfs.provided.aliasmap.leveldb.path=file:///path/to/leveldb/map/dingos.db -o org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.LevelDBFileRegionAliasMap` +`-Ddfs.provided.aliasmap.leveldb.path=file:///path/to/leveldb/map/dingos.db -b org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.LevelDBFileRegionAliasMap` as in the example above. Datanodes contact this alias map using the `org.apache.hadoop.hdfs.server.aliasmap.InMemoryAliasMapProtocol` protocol. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles.
Repository: hadoop Updated Branches: refs/heads/branch-3.2 4de3cf196 -> e185ae2d1 HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles. (cherry picked from commit 81f635f47f0737eb551bef1aa55afdf7b268253d) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e185ae2d Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e185ae2d Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e185ae2d Branch: refs/heads/branch-3.2 Commit: e185ae2d17e1ac4e432549fde077a5ee21041d8f Parents: 4de3cf1 Author: Yiqun Lin Authored: Thu Oct 4 10:30:30 2018 +0800 Committer: Yiqun Lin Committed: Thu Oct 4 11:04:39 2018 +0800 -- .../org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/e185ae2d/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java -- diff --git a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java index f6f670b..af781f5 100644 --- a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java +++ b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java @@ -38,6 +38,7 @@ import java.util.Map; @Consumes(MediaType.APPLICATION_JSON) @InterfaceAudience.Private public class KMSJSONReader implements MessageBodyReader { + private static final ObjectMapper MAPPER = new ObjectMapper(); @Override public boolean isReadable(Class type, Type genericType, @@ -51,7 +52,6 @@ public class KMSJSONReader implements MessageBodyReader { Annotation[] annotations, MediaType mediaType, MultivaluedMap httpHeaders, InputStream entityStream) throws IOException, WebApplicationException { -ObjectMapper mapper = new ObjectMapper(); -return mapper.readValue(entityStream, type); +return MAPPER.readValue(entityStream, type); } } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles.
Repository: hadoop Updated Branches: refs/heads/branch-2.9 54ef6e25b -> a3c564b01 HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles. (cherry picked from commit 81f635f47f0737eb551bef1aa55afdf7b268253d) (cherry picked from commit 7b88a57c379fe6eba7d685362e0bc756fdeff700) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a3c564b0 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a3c564b0 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a3c564b0 Branch: refs/heads/branch-2.9 Commit: a3c564b01e997b7911005ec3ad69fbd8e3a2af50 Parents: 54ef6e2 Author: Yiqun Lin Authored: Thu Oct 4 10:30:30 2018 +0800 Committer: Yiqun Lin Committed: Thu Oct 4 10:54:10 2018 +0800 -- .../org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/a3c564b0/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java -- diff --git a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java index d3e0064..a59e94c 100644 --- a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java +++ b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java @@ -36,6 +36,7 @@ import java.util.Map; @Consumes(MediaType.APPLICATION_JSON) @InterfaceAudience.Private public class KMSJSONReader implements MessageBodyReader { + private static final ObjectMapper MAPPER = new ObjectMapper(); @Override public boolean isReadable(Class type, Type genericType, @@ -48,7 +49,6 @@ public class KMSJSONReader implements MessageBodyReader { Annotation[] annotations, MediaType mediaType, MultivaluedMap httpHeaders, InputStream entityStream) throws IOException, WebApplicationException { -ObjectMapper mapper = new ObjectMapper(); -return mapper.readValue(entityStream, type); +return MAPPER.readValue(entityStream, type); } } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles.
Repository: hadoop Updated Branches: refs/heads/branch-2.8 d84958405 -> 94f4b5b9f HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles. (cherry picked from commit 81f635f47f0737eb551bef1aa55afdf7b268253d) (cherry picked from commit 7b88a57c379fe6eba7d685362e0bc756fdeff700) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/94f4b5b9 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/94f4b5b9 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/94f4b5b9 Branch: refs/heads/branch-2.8 Commit: 94f4b5b9f3a08100cd0759400bd3a52a23ef76fc Parents: d849584 Author: Yiqun Lin Authored: Thu Oct 4 10:30:30 2018 +0800 Committer: Yiqun Lin Committed: Thu Oct 4 10:52:13 2018 +0800 -- .../org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/94f4b5b9/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java -- diff --git a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java index d3e0064..a59e94c 100644 --- a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java +++ b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java @@ -36,6 +36,7 @@ import java.util.Map; @Consumes(MediaType.APPLICATION_JSON) @InterfaceAudience.Private public class KMSJSONReader implements MessageBodyReader { + private static final ObjectMapper MAPPER = new ObjectMapper(); @Override public boolean isReadable(Class type, Type genericType, @@ -48,7 +49,6 @@ public class KMSJSONReader implements MessageBodyReader { Annotation[] annotations, MediaType mediaType, MultivaluedMap httpHeaders, InputStream entityStream) throws IOException, WebApplicationException { -ObjectMapper mapper = new ObjectMapper(); -return mapper.readValue(entityStream, type); +return MAPPER.readValue(entityStream, type); } } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles.
Repository: hadoop Updated Branches: refs/heads/branch-2 65aaa1017 -> 7b88a57c3 HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles. (cherry picked from commit 81f635f47f0737eb551bef1aa55afdf7b268253d) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7b88a57c Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7b88a57c Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7b88a57c Branch: refs/heads/branch-2 Commit: 7b88a57c379fe6eba7d685362e0bc756fdeff700 Parents: 65aaa10 Author: Yiqun Lin Authored: Thu Oct 4 10:30:30 2018 +0800 Committer: Yiqun Lin Committed: Thu Oct 4 10:47:22 2018 +0800 -- .../org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/7b88a57c/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java -- diff --git a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java index d3e0064..a59e94c 100644 --- a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java +++ b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java @@ -36,6 +36,7 @@ import java.util.Map; @Consumes(MediaType.APPLICATION_JSON) @InterfaceAudience.Private public class KMSJSONReader implements MessageBodyReader { + private static final ObjectMapper MAPPER = new ObjectMapper(); @Override public boolean isReadable(Class type, Type genericType, @@ -48,7 +49,6 @@ public class KMSJSONReader implements MessageBodyReader { Annotation[] annotations, MediaType mediaType, MultivaluedMap httpHeaders, InputStream entityStream) throws IOException, WebApplicationException { -ObjectMapper mapper = new ObjectMapper(); -return mapper.readValue(entityStream, type); +return MAPPER.readValue(entityStream, type); } } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles.
Repository: hadoop Updated Branches: refs/heads/branch-3.0 b412bb224 -> d993a1fc3 HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles. (cherry picked from commit 81f635f47f0737eb551bef1aa55afdf7b268253d) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d993a1fc Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d993a1fc Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d993a1fc Branch: refs/heads/branch-3.0 Commit: d993a1fc3dfcf9944f4074d337b57db3bb348d60 Parents: b412bb2 Author: Yiqun Lin Authored: Thu Oct 4 10:30:30 2018 +0800 Committer: Yiqun Lin Committed: Thu Oct 4 10:36:49 2018 +0800 -- .../org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/d993a1fc/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java -- diff --git a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java index f6f670b..af781f5 100644 --- a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java +++ b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java @@ -38,6 +38,7 @@ import java.util.Map; @Consumes(MediaType.APPLICATION_JSON) @InterfaceAudience.Private public class KMSJSONReader implements MessageBodyReader { + private static final ObjectMapper MAPPER = new ObjectMapper(); @Override public boolean isReadable(Class type, Type genericType, @@ -51,7 +52,6 @@ public class KMSJSONReader implements MessageBodyReader { Annotation[] annotations, MediaType mediaType, MultivaluedMap httpHeaders, InputStream entityStream) throws IOException, WebApplicationException { -ObjectMapper mapper = new ObjectMapper(); -return mapper.readValue(entityStream, type); +return MAPPER.readValue(entityStream, type); } } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles.
Repository: hadoop Updated Branches: refs/heads/branch-3.1 58693c63d -> 1a890b17b HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles. (cherry picked from commit 81f635f47f0737eb551bef1aa55afdf7b268253d) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1a890b17 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1a890b17 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1a890b17 Branch: refs/heads/branch-3.1 Commit: 1a890b17b95d9069c7319553d8d060927bca750a Parents: 58693c6 Author: Yiqun Lin Authored: Thu Oct 4 10:30:30 2018 +0800 Committer: Yiqun Lin Committed: Thu Oct 4 10:35:16 2018 +0800 -- .../org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/1a890b17/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java -- diff --git a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java index f6f670b..af781f5 100644 --- a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java +++ b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java @@ -38,6 +38,7 @@ import java.util.Map; @Consumes(MediaType.APPLICATION_JSON) @InterfaceAudience.Private public class KMSJSONReader implements MessageBodyReader { + private static final ObjectMapper MAPPER = new ObjectMapper(); @Override public boolean isReadable(Class type, Type genericType, @@ -51,7 +52,6 @@ public class KMSJSONReader implements MessageBodyReader { Annotation[] annotations, MediaType mediaType, MultivaluedMap httpHeaders, InputStream entityStream) throws IOException, WebApplicationException { -ObjectMapper mapper = new ObjectMapper(); -return mapper.readValue(entityStream, type); +return MAPPER.readValue(entityStream, type); } } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles.
Repository: hadoop Updated Branches: refs/heads/trunk 39b35036b -> 81f635f47 HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/81f635f4 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/81f635f4 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/81f635f4 Branch: refs/heads/trunk Commit: 81f635f47f0737eb551bef1aa55afdf7b268253d Parents: 39b3503 Author: Yiqun Lin Authored: Thu Oct 4 10:30:30 2018 +0800 Committer: Yiqun Lin Committed: Thu Oct 4 10:30:30 2018 +0800 -- .../org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/81f635f4/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java -- diff --git a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java index f6f670b..af781f5 100644 --- a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java +++ b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONReader.java @@ -38,6 +38,7 @@ import java.util.Map; @Consumes(MediaType.APPLICATION_JSON) @InterfaceAudience.Private public class KMSJSONReader implements MessageBodyReader { + private static final ObjectMapper MAPPER = new ObjectMapper(); @Override public boolean isReadable(Class type, Type genericType, @@ -51,7 +52,6 @@ public class KMSJSONReader implements MessageBodyReader { Annotation[] annotations, MediaType mediaType, MultivaluedMap httpHeaders, InputStream entityStream) throws IOException, WebApplicationException { -ObjectMapper mapper = new ObjectMapper(); -return mapper.readValue(entityStream, type); +return MAPPER.readValue(entityStream, type); } } - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore.
Repository: hadoop Updated Branches: refs/heads/branch-3.1 c306da08e -> 65af98b58 HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore. (cherry picked from commit 5689355783de005ebc604f4403dc5129a286bfca) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/65af98b5 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/65af98b5 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/65af98b5 Branch: refs/heads/branch-3.1 Commit: 65af98b58a6cf66037a295e5ca951e31e472c8ce Parents: c306da0 Author: Yiqun Lin Authored: Tue Oct 2 09:43:14 2018 +0800 Committer: Yiqun Lin Committed: Tue Oct 2 09:46:23 2018 +0800 -- .../org/apache/hadoop/hdfs/DFSConfigKeys.java | 3 + .../hadoop/hdfs/server/datanode/DataNode.java | 2 +- .../datanode/fsdataset/impl/BlockPoolSlice.java | 177 +-- .../datanode/fsdataset/impl/FsDatasetImpl.java | 14 ++ .../datanode/fsdataset/impl/FsDatasetUtil.java | 30 ++-- .../datanode/fsdataset/impl/FsVolumeList.java | 5 +- .../datanode/fsdataset/impl/ReplicaMap.java | 25 +++ .../src/main/resources/hdfs-default.xml | 9 + .../fsdataset/impl/TestFsVolumeList.java| 64 ++- 9 files changed, 300 insertions(+), 29 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/65af98b5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java index 97bb469..aa5e758 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java @@ -359,6 +359,9 @@ public class DFSConfigKeys extends CommonConfigurationKeys { public static final longDFS_CONTENT_SUMMARY_SLEEP_MICROSEC_DEFAULT = 500; public static final String DFS_DATANODE_FAILED_VOLUMES_TOLERATED_KEY = "dfs.datanode.failed.volumes.tolerated"; public static final int DFS_DATANODE_FAILED_VOLUMES_TOLERATED_DEFAULT = 0; + public static final String + DFS_DATANODE_VOLUMES_REPLICA_ADD_THREADPOOL_SIZE_KEY = + "dfs.datanode.volumes.replica-add.threadpool.size"; public static final String DFS_DATANODE_SYNCONCLOSE_KEY = "dfs.datanode.synconclose"; public static final boolean DFS_DATANODE_SYNCONCLOSE_DEFAULT = false; public static final String DFS_DATANODE_SOCKET_REUSE_KEEPALIVE_KEY = "dfs.datanode.socket.reuse.keepalive"; http://git-wip-us.apache.org/repos/asf/hadoop/blob/65af98b5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java index ade2b11..787f42c 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java @@ -1695,7 +1695,7 @@ public class DataNode extends ReconfigurableBase return blockPoolManager.get(bpid); } - int getBpOsCount() { + public int getBpOsCount() { return blockPoolManager.getAllNamenodeThreads().size(); } http://git-wip-us.apache.org/repos/asf/hadoop/blob/65af98b5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java index 3f9de78..d7e56fd 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java @@ -28,8 +28,19 @@ import java.io.InputStream; import java.io.OutputStreamWriter; import java.io.RandomAccessFile; import java.io.Writer; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.Comparator; import java.util.Iterator; +import java.util.List; +import java.util.Queue; import java.util.Sca
hadoop git commit: HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore.
Repository: hadoop Updated Branches: refs/heads/trunk f6c5ef990 -> 568935578 HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/56893557 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/56893557 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/56893557 Branch: refs/heads/trunk Commit: 5689355783de005ebc604f4403dc5129a286bfca Parents: f6c5ef9 Author: Yiqun Lin Authored: Tue Oct 2 09:43:14 2018 +0800 Committer: Yiqun Lin Committed: Tue Oct 2 09:43:14 2018 +0800 -- .../org/apache/hadoop/hdfs/DFSConfigKeys.java | 3 + .../hadoop/hdfs/server/datanode/DataNode.java | 2 +- .../datanode/fsdataset/impl/BlockPoolSlice.java | 177 +-- .../datanode/fsdataset/impl/FsDatasetImpl.java | 14 ++ .../datanode/fsdataset/impl/FsDatasetUtil.java | 30 ++-- .../datanode/fsdataset/impl/FsVolumeList.java | 5 +- .../datanode/fsdataset/impl/ReplicaMap.java | 25 +++ .../src/main/resources/hdfs-default.xml | 9 + .../fsdataset/impl/TestFsVolumeList.java| 64 ++- 9 files changed, 300 insertions(+), 29 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/56893557/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java index a7e7b9b..d8024dc 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java @@ -365,6 +365,9 @@ public class DFSConfigKeys extends CommonConfigurationKeys { public static final longDFS_CONTENT_SUMMARY_SLEEP_MICROSEC_DEFAULT = 500; public static final String DFS_DATANODE_FAILED_VOLUMES_TOLERATED_KEY = "dfs.datanode.failed.volumes.tolerated"; public static final int DFS_DATANODE_FAILED_VOLUMES_TOLERATED_DEFAULT = 0; + public static final String + DFS_DATANODE_VOLUMES_REPLICA_ADD_THREADPOOL_SIZE_KEY = + "dfs.datanode.volumes.replica-add.threadpool.size"; public static final String DFS_DATANODE_SYNCONCLOSE_KEY = "dfs.datanode.synconclose"; public static final boolean DFS_DATANODE_SYNCONCLOSE_DEFAULT = false; public static final String DFS_DATANODE_SOCKET_REUSE_KEEPALIVE_KEY = "dfs.datanode.socket.reuse.keepalive"; http://git-wip-us.apache.org/repos/asf/hadoop/blob/56893557/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java index c980395..270e30b 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java @@ -1695,7 +1695,7 @@ public class DataNode extends ReconfigurableBase return blockPoolManager.get(bpid); } - int getBpOsCount() { + public int getBpOsCount() { return blockPoolManager.getAllNamenodeThreads().size(); } http://git-wip-us.apache.org/repos/asf/hadoop/blob/56893557/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java index 2adfb6b..b9b581f 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java @@ -28,8 +28,19 @@ import java.io.InputStream; import java.io.OutputStreamWriter; import java.io.RandomAccessFile; import java.io.Writer; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.Comparator; import java.util.Iterator; +import java.util.List; +import java.util.Queue; import java.util.Scanner; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.conc
hadoop git commit: HADOOP-15742. Log if ipc backoff is enabled in CallQueueManager. Contributed by Ryan Wu.
Repository: hadoop Updated Branches: refs/heads/trunk 281c192e7 -> ee051ef9f HADOOP-15742. Log if ipc backoff is enabled in CallQueueManager. Contributed by Ryan Wu. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ee051ef9 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ee051ef9 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ee051ef9 Branch: refs/heads/trunk Commit: ee051ef9fec1fddb612aa1feae9fd3df7091354f Parents: 281c192 Author: Yiqun Lin Authored: Tue Sep 18 11:10:33 2018 +0800 Committer: Yiqun Lin Committed: Tue Sep 18 11:10:33 2018 +0800 -- .../src/main/java/org/apache/hadoop/ipc/CallQueueManager.java | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/ee051ef9/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java -- diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java index d1bd180..29649a6 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java @@ -81,8 +81,9 @@ public class CallQueueManager this.clientBackOffEnabled = clientBackOffEnabled; this.putRef = new AtomicReference>(bq); this.takeRef = new AtomicReference>(bq); -LOG.info("Using callQueue: " + backingClass + " queueCapacity: " + -maxQueueSize + " scheduler: " + schedulerClass); +LOG.info("Using callQueue: {}, queueCapacity: {}, " + +"scheduler: {}, ipcBackoff: {}.", +backingClass, maxQueueSize, schedulerClass, clientBackOffEnabled); } @VisibleForTesting // only! - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13884. Improve the description of the setting dfs.image.compress. Contributed by Ryan Wu.
Repository: hadoop Updated Branches: refs/heads/trunk eef3bafae -> 0da49642f HDFS-13884. Improve the description of the setting dfs.image.compress. Contributed by Ryan Wu. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0da49642 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0da49642 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0da49642 Branch: refs/heads/trunk Commit: 0da49642fc1eb71997b1aa268583c1ba09a16687 Parents: eef3baf Author: Yiqun Lin Authored: Mon Sep 10 13:57:36 2018 +0800 Committer: Yiqun Lin Committed: Mon Sep 10 13:57:36 2018 +0800 -- .../hadoop-hdfs/src/main/resources/hdfs-default.xml| 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/0da49642/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml -- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml index 5f115ec..1573582 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml @@ -1284,7 +1284,11 @@ dfs.image.compress false - Should the dfs image be compressed? + When this value is true, the dfs image will be compressed. +Enabling this will be very helpful if dfs image is large since it can +avoid consuming a lot of network bandwidth when SBN uploads a new dfs +image to ANN. The compressed codec is specified by the setting +dfs.image.compression.codec. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13815. RBF: Add check to order command. Contributed by Ranith Sardar.
Repository: hadoop Updated Branches: refs/heads/branch-3.1 2d68708a1 -> c898757f5 HDFS-13815. RBF: Add check to order command. Contributed by Ranith Sardar. (cherry picked from commit 9315db5f5da09c2ef86be168465c16932afa2d85) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c898757f Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c898757f Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c898757f Branch: refs/heads/branch-3.1 Commit: c898757f5541a523b12f2e5cfb504d624da13e1b Parents: 2d68708 Author: Yiqun Lin Authored: Wed Sep 5 23:33:27 2018 +0800 Committer: Yiqun Lin Committed: Wed Sep 5 23:35:30 2018 +0800 -- .../hdfs/tools/federation/RouterAdmin.java | 10 .../federation/router/TestRouterAdminCLI.java | 57 +++- 2 files changed, 66 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/c898757f/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java index ef8d7c1..0a681e9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java @@ -263,10 +263,14 @@ public class RouterAdmin extends Configured implements Tool { if ("-add".equals(cmd)) { if (addMount(argv, i)) { System.out.println("Successfully added mount point " + argv[i]); +} else { + exitCode = -1; } } else if ("-update".equals(cmd)) { if (updateMount(argv, i)) { System.out.println("Successfully updated mount point " + argv[i]); +} else { + exitCode = -1; } } else if ("-rm".equals(cmd)) { if (removeMount(argv[i])) { @@ -369,6 +373,9 @@ public class RouterAdmin extends Configured implements Tool { i++; short modeValue = Short.parseShort(parameters[i], 8); mode = new FsPermission(modeValue); + } else { +printUsage("-add"); +return false; } i++; @@ -521,6 +528,9 @@ public class RouterAdmin extends Configured implements Tool { i++; short modeValue = Short.parseShort(parameters[i], 8); mode = new FsPermission(modeValue); + } else { +printUsage("-update"); +return false; } i++; http://git-wip-us.apache.org/repos/asf/hadoop/blob/c898757f/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java index fa29cd9..d968c60 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java @@ -224,6 +224,24 @@ public class TestRouterAdminCLI { testAddOrderMountTable(DestinationOrder.HASH_ALL); } + @Test + public void testAddOrderErrorMsg() throws Exception { +DestinationOrder order = DestinationOrder.HASH; +final String mnt = "/newAdd1" + order; +final String nsId = "ns0,ns1"; +final String dest = "/changAdd"; + +String[] argv1 = new String[] {"-add", mnt, nsId, dest, "-order", +order.toString()}; +assertEquals(0, ToolRunner.run(admin, argv1)); + +// Add the order with wrong command +String[] argv = new String[] {"-add", mnt, nsId, dest, "-orde", +order.toString()}; +assertEquals(-1, ToolRunner.run(admin, argv)); + + } + private void testAddOrderMountTable(DestinationOrder order) throws Exception { final String mnt = "/" + order; @@ -403,7 +421,7 @@ public class TestRouterAdminCLI { argv = new String[] {"-add", "/testpath2-2", "ns0", "/testdir2-2", "-owner", TEST_USER, "-group", TEST_USER, "-mode", "0255"}; assertEquals(0, ToolRunner.run(admin, argv)); -verifyExecutionResult("/testpath2-2", false, 0, 0); +verifyExecutionResult("/testpath2-2", false, -1, 0); // set mount table entry with read and write permissio
hadoop git commit: HDFS-13815. RBF: Add check to order command. Contributed by Ranith Sardar.
Repository: hadoop Updated Branches: refs/heads/trunk df0d61e3a -> 9315db5f5 HDFS-13815. RBF: Add check to order command. Contributed by Ranith Sardar. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9315db5f Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9315db5f Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9315db5f Branch: refs/heads/trunk Commit: 9315db5f5da09c2ef86be168465c16932afa2d85 Parents: df0d61e Author: Yiqun Lin Authored: Wed Sep 5 23:33:27 2018 +0800 Committer: Yiqun Lin Committed: Wed Sep 5 23:33:27 2018 +0800 -- .../hdfs/tools/federation/RouterAdmin.java | 10 .../federation/router/TestRouterAdminCLI.java | 57 +++- 2 files changed, 66 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/9315db5f/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java index ef8d7c1..0a681e9 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java @@ -263,10 +263,14 @@ public class RouterAdmin extends Configured implements Tool { if ("-add".equals(cmd)) { if (addMount(argv, i)) { System.out.println("Successfully added mount point " + argv[i]); +} else { + exitCode = -1; } } else if ("-update".equals(cmd)) { if (updateMount(argv, i)) { System.out.println("Successfully updated mount point " + argv[i]); +} else { + exitCode = -1; } } else if ("-rm".equals(cmd)) { if (removeMount(argv[i])) { @@ -369,6 +373,9 @@ public class RouterAdmin extends Configured implements Tool { i++; short modeValue = Short.parseShort(parameters[i], 8); mode = new FsPermission(modeValue); + } else { +printUsage("-add"); +return false; } i++; @@ -521,6 +528,9 @@ public class RouterAdmin extends Configured implements Tool { i++; short modeValue = Short.parseShort(parameters[i], 8); mode = new FsPermission(modeValue); + } else { +printUsage("-update"); +return false; } i++; http://git-wip-us.apache.org/repos/asf/hadoop/blob/9315db5f/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java -- diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java index fa29cd9..d968c60 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java +++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java @@ -224,6 +224,24 @@ public class TestRouterAdminCLI { testAddOrderMountTable(DestinationOrder.HASH_ALL); } + @Test + public void testAddOrderErrorMsg() throws Exception { +DestinationOrder order = DestinationOrder.HASH; +final String mnt = "/newAdd1" + order; +final String nsId = "ns0,ns1"; +final String dest = "/changAdd"; + +String[] argv1 = new String[] {"-add", mnt, nsId, dest, "-order", +order.toString()}; +assertEquals(0, ToolRunner.run(admin, argv1)); + +// Add the order with wrong command +String[] argv = new String[] {"-add", mnt, nsId, dest, "-orde", +order.toString()}; +assertEquals(-1, ToolRunner.run(admin, argv)); + + } + private void testAddOrderMountTable(DestinationOrder order) throws Exception { final String mnt = "/" + order; @@ -403,7 +421,7 @@ public class TestRouterAdminCLI { argv = new String[] {"-add", "/testpath2-2", "ns0", "/testdir2-2", "-owner", TEST_USER, "-group", TEST_USER, "-mode", "0255"}; assertEquals(0, ToolRunner.run(admin, argv)); -verifyExecutionResult("/testpath2-2", false, 0, 0); +verifyExecutionResult("/testpath2-2", false, -1, 0); // set mount table entry with read and write permission argv = new String[] {"-add", "/testpath2-3", "ns0", "/testdir2-3", @@ -88