[GitHub] [hadoop-ozone] nandakumar131 commented on issue #313: HDDS-2242. Avoid unnecessary rpc needed to discover the pipeline leader.
nandakumar131 commented on issue #313: HDDS-2242. Avoid unnecessary rpc needed to discover the pipeline leader. URL: https://github.com/apache/hadoop-ozone/pull/313#issuecomment-565333775 > > The code will become a lot simpler if the leader node in `Pipeline` is represented as `DatanodeDetails` instead of just `UUID`. > > Hi, @nandakumar131 this was changed based on your suggestion in HDDS-1868 :-) > Although it isn't affecting performance, can we refactor later, this patch was meant as downstream work and I was hoping not to change proto structure for it. @swagle Even in HDDS-1868 the suggestion was to use `DatanodeDetails` instead of just `UUID`. We can do it in a follow-up jira. I will merge this shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r357511301
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
##
@@ -163,6 +177,18 @@ public synchronized Pipeline
createPipeline(ReplicationType type,
metrics.incNumPipelineCreated();
metrics.createPerPipelineMetrics(pipeline);
}
+ Pipeline overlapPipeline = RatisPipelineUtils
+ .checkPipelineContainSameDatanodes(stateManager, pipeline);
+ if (overlapPipeline != null) {
+metrics.incNumPipelineContainSameDatanodes();
+//TODO remove until pipeline allocation is proved equally distributed.
+LOG.info("Pipeline: " + pipeline.getId().toString() +
Review comment:
Talked offline. OK to have this at INFO level for now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r357509565
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -100,4 +105,45 @@ static void destroyPipeline(DatanodeDetails dn,
PipelineID pipelineID,
true, p.getId());
}
}
+
+ static int encodeNodeIdsOfFactorThreePipeline(List nodes) {
+if (nodes.size() != HddsProtos.ReplicationFactor.THREE.getNumber()) {
+ return 0;
+}
+List nodeIds = nodes.stream()
+.map(DatanodeDetails::getUuid).distinct()
+.collect(Collectors.toList());
+nodeIds.sort(new ComparableComparator());
+// Only for Factor THREE pipeline.
+return new HashCodeBuilder()
+.append(nodeIds.get(0).toString())
+.append(nodeIds.get(1).toString())
+.append(nodeIds.get(2).toString())
+.toHashCode();
+ }
+
+ /**
+ * Return first existed pipeline which share the same set of datanodes
+ * with the input pipeline.
+ * @param stateManager PipelineStateManager
+ * @param pipeline input pipeline
+ * @return first matched pipeline
+ */
+ static Pipeline checkPipelineContainSameDatanodes(
Review comment:
Both should work just non-static seems cleaner. But if there are too much
additional work to fix the unit tests. I'm OK with current approach.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #341: HDDS-2414. Simplify robot tests with removing output greps
adoroszlai commented on issue #341: HDDS-2414. Simplify robot tests with removing output greps URL: https://github.com/apache/hadoop-ozone/pull/341#issuecomment-565327920 @elek please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #342: HDDS-2709. Maven property skipShade should not skip ozonefs compilation
adoroszlai commented on issue #342: HDDS-2709. Maven property skipShade should not skip ozonefs compilation URL: https://github.com/apache/hadoop-ozone/pull/342#issuecomment-565327979 @elek please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #348: HDDS-2035 Implement datanode level CLI to reveal pipeline relation.
adoroszlai commented on a change in pull request #348: HDDS-2035 Implement
datanode level CLI to reveal pipeline relation.
URL: https://github.com/apache/hadoop-ozone/pull/348#discussion_r357503054
##
File path:
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/ListInfoSubcommand.java
##
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.scm.cli.datanode;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.stream.Collectors;
+
+/**
+ * Handler of list datanodes info command.
+ */
+@CommandLine.Command(
+name = "list",
+description = "List info of datanodes",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class ListInfoSubcommand implements Callable {
+
+ @CommandLine.ParentCommand
+ private DatanodeCommands parent;
+
+ @CommandLine.Option(names = {"-ip", "--byIp"},
+ description = "Show info by ip address.",
+ defaultValue = "",
+ required = false)
+ private String ipaddress;
+
+ @CommandLine.Option(names = {"-id", "--byUuid"},
+ description = "Show info by datanode UUID.",
+ defaultValue = "",
+ required = false)
+ private String uuid;
+
+ private List pipelines;
+
+
+ @Override
+ public Void call() throws Exception {
+try (ScmClient scmClient = parent.getParent().createScmClient()) {
+ pipelines = scmClient.listPipelines();
+ if (isNullOrEmpty(ipaddress) && isNullOrEmpty(uuid)) {
+getAllNodes(scmClient).stream().forEach(p -> printDatanodeInfo(p));
+ } else {
+getAllNodes(scmClient).stream().filter(
+p -> ((isNullOrEmpty(ipaddress) ||
+(p.getIpAddress().compareToIgnoreCase(ipaddress) == 0))
+&& (isNullOrEmpty(uuid) ||
+(p.getUuid().equals(uuid)
+.forEach(p -> printDatanodeInfo(p));
+ }
Review comment:
Structuring this as below would reduce duplication, and make it easier to
add further filters if needed:
```
Stream<...> stream = getAllNodes(scmClient).stream();
if (!isNullOrEmpty(ipaddress)) {
stream = stream.filter(...);
}
if (!isNullOrEmpty(uuid)) {
stream = stream.filter(...);
}
stream.forEach(...);
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #348: HDDS-2035 Implement datanode level CLI to reveal pipeline relation.
adoroszlai commented on a change in pull request #348: HDDS-2035 Implement
datanode level CLI to reveal pipeline relation.
URL: https://github.com/apache/hadoop-ozone/pull/348#discussion_r357502346
##
File path:
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/ListInfoSubcommand.java
##
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.scm.cli.datanode;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.stream.Collectors;
+
+/**
+ * Handler of list datanodes info command.
+ */
+@CommandLine.Command(
+name = "list",
+description = "List info of datanodes",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class ListInfoSubcommand implements Callable {
+
+ @CommandLine.ParentCommand
+ private DatanodeCommands parent;
+
+ @CommandLine.Option(names = {"-ip", "--byIp"},
+ description = "Show info by ip address.",
+ defaultValue = "",
+ required = false)
+ private String ipaddress;
+
+ @CommandLine.Option(names = {"-id", "--byUuid"},
Review comment:
Similarly, suggesting only `--uuid`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #348: HDDS-2035 Implement datanode level CLI to reveal pipeline relation.
adoroszlai commented on a change in pull request #348: HDDS-2035 Implement
datanode level CLI to reveal pipeline relation.
URL: https://github.com/apache/hadoop-ozone/pull/348#discussion_r357507729
##
File path:
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/ListInfoSubcommand.java
##
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.scm.cli.datanode;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.stream.Collectors;
+
+/**
+ * Handler of list datanodes info command.
+ */
+@CommandLine.Command(
+name = "list",
+description = "List info of datanodes",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class ListInfoSubcommand implements Callable {
+
+ @CommandLine.ParentCommand
+ private DatanodeCommands parent;
+
+ @CommandLine.Option(names = {"-ip", "--byIp"},
+ description = "Show info by ip address.",
+ defaultValue = "",
+ required = false)
+ private String ipaddress;
+
+ @CommandLine.Option(names = {"-id", "--byUuid"},
+ description = "Show info by datanode UUID.",
+ defaultValue = "",
+ required = false)
+ private String uuid;
+
+ private List pipelines;
+
+
+ @Override
+ public Void call() throws Exception {
+try (ScmClient scmClient = parent.getParent().createScmClient()) {
+ pipelines = scmClient.listPipelines();
+ if (isNullOrEmpty(ipaddress) && isNullOrEmpty(uuid)) {
+getAllNodes(scmClient).stream().forEach(p -> printDatanodeInfo(p));
+ } else {
+getAllNodes(scmClient).stream().filter(
+p -> ((isNullOrEmpty(ipaddress) ||
+(p.getIpAddress().compareToIgnoreCase(ipaddress) == 0))
+&& (isNullOrEmpty(uuid) ||
+(p.getUuid().equals(uuid)
+.forEach(p -> printDatanodeInfo(p));
+ }
+ return null;
+}
+ }
+
+ private List getAllNodes(ScmClient scmClient)
+ throws IOException {
+List nodes = scmClient.queryNode(
+HddsProtos.NodeState.HEALTHY, HddsProtos.QueryScope.CLUSTER, "");
+List datanodes = new ArrayList<>(nodes.size());
+nodes.stream().forEach(
+p -> datanodes.add(DatanodeDetails.getFromProtoBuf(p.getNodeID(;
+return datanodes;
+ }
+
+ private void printDatanodeInfo(DatanodeDetails datanode) {
+if (pipelines.isEmpty()) {
+ System.out.println("Datanode: " + datanode.getUuid().toString() +
+ " (" + datanode.getIpAddress() + "/"
+ + datanode.getHostName() + "). \n No pipeline created.");
+ return;
+}
+List relatedPipelines = pipelines.stream().filter(
+p -> p.getNodes().contains(datanode)).collect(Collectors.toList());
+StringBuilder pipelineList = new StringBuilder()
+.append("Related pipelines:\n");
+relatedPipelines.stream().forEach(
+p -> pipelineList.append(p.getId().getId().toString())
+.append("/").append(p.getFactor().toString()).append("/")
+.append(p.getType().toString()).append("/")
+.append(p.getPipelineState().toString()).append("/")
+.append(datanode.getUuid().equals(p.getLeaderId()) ?
+"Leader" : "Follower")
+.append(System.getProperty("line.separator")));
+System.out.println("Datanode: " + datanode.getUuid().toString() +
+" (" + datanode.getIpAddress() + "/"
++ datanode.getHostName() + "/" + relatedPipelines.size() +
+" pipelines). \n" + pipelineList);
Review comment:
Even if there are pipelines, the datanode may happen to be not part of any.
Handling "no related pipelines" here would also allow removing the "No pipeline
created" special case above.
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #348: HDDS-2035 Implement datanode level CLI to reveal pipeline relation.
adoroszlai commented on a change in pull request #348: HDDS-2035 Implement
datanode level CLI to reveal pipeline relation.
URL: https://github.com/apache/hadoop-ozone/pull/348#discussion_r357500392
##
File path:
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/ListInfoSubcommand.java
##
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.scm.cli.datanode;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.stream.Collectors;
+
+/**
+ * Handler of list datanodes info command.
+ */
+@CommandLine.Command(
+name = "list",
+description = "List info of datanodes",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class ListInfoSubcommand implements Callable {
Review comment:
`ListDatanodesSubcommand` might better reflect what this subcommand is about.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #348: HDDS-2035 Implement datanode level CLI to reveal pipeline relation.
adoroszlai commented on a change in pull request #348: HDDS-2035 Implement
datanode level CLI to reveal pipeline relation.
URL: https://github.com/apache/hadoop-ozone/pull/348#discussion_r357502263
##
File path:
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/ListInfoSubcommand.java
##
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.scm.cli.datanode;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.stream.Collectors;
+
+/**
+ * Handler of list datanodes info command.
+ */
+@CommandLine.Command(
+name = "list",
+description = "List info of datanodes",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class ListInfoSubcommand implements Callable {
+
+ @CommandLine.ParentCommand
+ private DatanodeCommands parent;
+
+ @CommandLine.Option(names = {"-ip", "--byIp"},
Review comment:
I think `-` should be reserved for single-char options. I suggest adding
only `--ip` for simplicity. It's only slightly longer than `-ip`, but let's us
avoid confusion with combined single-char options.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #348: HDDS-2035 Implement datanode level CLI to reveal pipeline relation.
adoroszlai commented on a change in pull request #348: HDDS-2035 Implement
datanode level CLI to reveal pipeline relation.
URL: https://github.com/apache/hadoop-ozone/pull/348#discussion_r357505137
##
File path:
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/ListInfoSubcommand.java
##
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.scm.cli.datanode;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.stream.Collectors;
+
+/**
+ * Handler of list datanodes info command.
+ */
+@CommandLine.Command(
+name = "list",
+description = "List info of datanodes",
+mixinStandardHelpOptions = true,
+versionProvider = HddsVersionProvider.class)
+public class ListInfoSubcommand implements Callable {
+
+ @CommandLine.ParentCommand
+ private DatanodeCommands parent;
+
+ @CommandLine.Option(names = {"-ip", "--byIp"},
+ description = "Show info by ip address.",
+ defaultValue = "",
+ required = false)
+ private String ipaddress;
+
+ @CommandLine.Option(names = {"-id", "--byUuid"},
+ description = "Show info by datanode UUID.",
+ defaultValue = "",
+ required = false)
+ private String uuid;
+
+ private List pipelines;
+
+
+ @Override
+ public Void call() throws Exception {
+try (ScmClient scmClient = parent.getParent().createScmClient()) {
+ pipelines = scmClient.listPipelines();
+ if (isNullOrEmpty(ipaddress) && isNullOrEmpty(uuid)) {
+getAllNodes(scmClient).stream().forEach(p -> printDatanodeInfo(p));
+ } else {
+getAllNodes(scmClient).stream().filter(
+p -> ((isNullOrEmpty(ipaddress) ||
+(p.getIpAddress().compareToIgnoreCase(ipaddress) == 0))
+&& (isNullOrEmpty(uuid) ||
+(p.getUuid().equals(uuid)
+.forEach(p -> printDatanodeInfo(p));
+ }
+ return null;
+}
+ }
+
+ private List getAllNodes(ScmClient scmClient)
+ throws IOException {
+List nodes = scmClient.queryNode(
+HddsProtos.NodeState.HEALTHY, HddsProtos.QueryScope.CLUSTER, "");
+List datanodes = new ArrayList<>(nodes.size());
+nodes.stream().forEach(
+p -> datanodes.add(DatanodeDetails.getFromProtoBuf(p.getNodeID(;
+return datanodes;
Review comment:
Please try to avoid using `forEach` to fill an existing list, prefer
collectors instead, eg. `stream().map(...).collect(toList())`. It gives the
JDK more chance to improve.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2722) Let ChunkManager read/write ChunkBuffer instead of ByteBuffer
[
https://issues.apache.org/jira/browse/HDDS-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-2722:
---
Status: Patch Available (was: In Progress)
> Let ChunkManager read/write ChunkBuffer instead of ByteBuffer
> -
>
> Key: HDDS-2722
> URL: https://issues.apache.org/jira/browse/HDDS-2722
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Change {{ChunkManager}} read/write methods to accept/return {{ChunkBuffer}}
> instead of {{ByteBuffer}}. This allows seamlessly passing multiple buffers
> without further interface change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2722) Let ChunkManager read/write ChunkBuffer instead of ByteBuffer
[
https://issues.apache.org/jira/browse/HDDS-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2722:
-
Labels: pull-request-available (was: )
> Let ChunkManager read/write ChunkBuffer instead of ByteBuffer
> -
>
> Key: HDDS-2722
> URL: https://issues.apache.org/jira/browse/HDDS-2722
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
>
> Change {{ChunkManager}} read/write methods to accept/return {{ChunkBuffer}}
> instead of {{ByteBuffer}}. This allows seamlessly passing multiple buffers
> without further interface change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #352: HDDS-2722. Let ChunkManager read/write ChunkBuffer instead of ByteBuffer
adoroszlai opened a new pull request #352: HDDS-2722. Let ChunkManager read/write ChunkBuffer instead of ByteBuffer URL: https://github.com/apache/hadoop-ozone/pull/352 ## What changes were proposed in this pull request? Change `ChunkManager` read/write methods to accept/return `ChunkBuffer` instead of `ByteBuffer`. This allows seamlessly passing multiple buffers without further interface change. https://issues.apache.org/jira/browse/HDDS-2722 ## How was this patch tested? Added unit test for new functionality in `ChunkBuffer`. The rest is covered by existing tests. https://github.com/adoroszlai/hadoop-ozone/runs/346822935 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1590) Add per pipeline metrics
[ https://issues.apache.org/jira/browse/HDDS-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao reassigned HDDS-1590: Assignee: Li Cheng > Add per pipeline metrics > > > Key: HDDS-1590 > URL: https://issues.apache.org/jira/browse/HDDS-1590 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Siddharth Wagle >Assignee: Li Cheng >Priority: Major > > Add metrics on per pipeline basis: > - bytes read/written > - container metrics (state changes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1575) Add support for storage policies to Pipelines
[ https://issues.apache.org/jira/browse/HDDS-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-1575: - Parent: (was: HDDS-1564) Issue Type: Improvement (was: Sub-task) > Add support for storage policies to Pipelines > - > > Key: HDDS-1575 > URL: https://issues.apache.org/jira/browse/HDDS-1575 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Siddharth Wagle >Priority: Major > > The Pipeline level storage policy can be thought of segregating write > bandwidth for high throughput writes or random reads use cases when SSDs, > NVMe or RAM disks are involved by not having to figure out write pipelines on > the fly. The Datanode would need to read the pipeline policy and route write > to appropriate storage volumes. An Ozone mover can be provided for the > archival data use case. > Additionally, it makes sense to permeate the policy information to > Containers, this will help when allocateBlock call for a key with storage > policy set to ALL_SSD, to be routed to an open container on a Pipeline with > the same policy setting. The caveat is that at least one open pipeline needs > to be maintained at all times in order to support storage the policy setting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1575) Add support for storage policies to Pipelines
[ https://issues.apache.org/jira/browse/HDDS-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995366#comment-16995366 ] Xiaoyu Yao commented on HDDS-1575: -- Move this out of multi-raft support. This can be implemented without multi-raft. > Add support for storage policies to Pipelines > - > > Key: HDDS-1575 > URL: https://issues.apache.org/jira/browse/HDDS-1575 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Siddharth Wagle >Priority: Major > > The Pipeline level storage policy can be thought of segregating write > bandwidth for high throughput writes or random reads use cases when SSDs, > NVMe or RAM disks are involved by not having to figure out write pipelines on > the fly. The Datanode would need to read the pipeline policy and route write > to appropriate storage volumes. An Ozone mover can be provided for the > archival data use case. > Additionally, it makes sense to permeate the policy information to > Containers, this will help when allocateBlock call for a key with storage > policy set to ALL_SSD, to be routed to an open container on a Pipeline with > the same policy setting. The caveat is that at least one open pipeline needs > to be maintained at all times in order to support storage the policy setting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on issue #331: HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro…
avijayanhwx commented on issue #331: HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro… URL: https://github.com/apache/hadoop-ozone/pull/331#issuecomment-565305120 The acceptance test failures are due to this patch. I am looking into it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2176) Add new pipeline state “CLOSING” and new CLOSE_PIPELINE_STATUS command
[ https://issues.apache.org/jira/browse/HDDS-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-2176: - Parent: (was: HDDS-1564) Issue Type: Improvement (was: Sub-task) > Add new pipeline state “CLOSING” and new CLOSE_PIPELINE_STATUS command > -- > > Key: HDDS-2176 > URL: https://issues.apache.org/jira/browse/HDDS-2176 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > > Currrent pipeline has 3 states, ALLOCATED, OPEN and CLOSED. > When create pipeline command is sent out to datanodes from SCM, this pipeline > is marked as ALLOCATED in SCM. Once SCM received all 3 datanodes > confirmation of pipelinee creation, SCM will change pipeline's state from > ALLOCATED to OPEN. > Close pipeline process is similar. Add a new CLOSING state to pipeline. When > close pipeline command is sent out to datanodes, pipeline will be marked as > CLOSING. When all 3 datanode confirmed, change pipeline state from CLOSING > to CLOSED. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2034) Async RATIS pipeline creation and destroy through heartbeat commands
[ https://issues.apache.org/jira/browse/HDDS-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-2034: - Parent: (was: HDDS-1564) Issue Type: New Feature (was: Sub-task) > Async RATIS pipeline creation and destroy through heartbeat commands > > > Key: HDDS-2034 > URL: https://issues.apache.org/jira/browse/HDDS-2034 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Sammi Chen >Assignee: Sammi Chen >Priority: Major > Labels: pull-request-available > Fix For: 0.4.2 > > Time Spent: 14h > Remaining Estimate: 0h > > Currently, pipeline creation and destroy are synchronous operations. SCM > directly connect to each datanode of the pipeline through gRPC channel to > create the pipeline to destroy the pipeline. > This task is to remove the gRPC channel, send pipeline creation and destroy > action through heartbeat command to each datanode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-2036) Multi-raft support on single datanode integration test
[ https://issues.apache.org/jira/browse/HDDS-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995363#comment-16995363 ] Xiaoyu Yao edited comment on HDDS-2036 at 12/13/19 5:16 AM: The test change is covered by HDDS-1569. was (Author: xyao): The test change is covered by HDDS-1576. > Multi-raft support on single datanode integration test > -- > > Key: HDDS-2036 > URL: https://issues.apache.org/jira/browse/HDDS-2036 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Priority: Major > > Improve MiniOzoneCluster to support multi-raft group -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2036) Multi-raft support on single datanode integration test
[ https://issues.apache.org/jira/browse/HDDS-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-2036. -- Resolution: Fixed The test change is covered by HDDS-1576. > Multi-raft support on single datanode integration test > -- > > Key: HDDS-2036 > URL: https://issues.apache.org/jira/browse/HDDS-2036 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Priority: Major > > Improve MiniOzoneCluster to support multi-raft group -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1573) Add scrubber metrics and pipeline metrics
[ https://issues.apache.org/jira/browse/HDDS-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-1573. -- Resolution: Implemented There metrics have been implemented as part of the feature. > Add scrubber metrics and pipeline metrics > - > > Key: HDDS-1573 > URL: https://issues.apache.org/jira/browse/HDDS-1573 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Siddharth Wagle >Assignee: Li Cheng >Priority: Major > > - Add metrics for how many pipelines per datanode > - Add metrics for pipelines that are chosen by the scrubber > - Add metrics for pipelines that are in violation -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1590) Add per pipeline metrics
[ https://issues.apache.org/jira/browse/HDDS-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995357#comment-16995357 ] Xiaoyu Yao commented on HDDS-1590: -- TODO: add SCM per pipeline bytes write metrics. > Add per pipeline metrics > > > Key: HDDS-1590 > URL: https://issues.apache.org/jira/browse/HDDS-1590 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Siddharth Wagle >Priority: Major > > Add metrics on per pipeline basis: > - bytes read/written > - container metrics (state changes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #322: HDDS-2684. Refactor common test utilities to hadoop-hdds/common
adoroszlai commented on issue #322: HDDS-2684. Refactor common test utilities to hadoop-hdds/common URL: https://github.com/apache/hadoop-ozone/pull/322#issuecomment-565287537 Thanks @arp7 for reviewing and committing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2684) Refactor common test utilities to hadoop-hdds/common
[
https://issues.apache.org/jira/browse/HDDS-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDDS-2684:
Fix Version/s: 0.5.0
Target Version/s: (was: 0.5.0)
Resolution: Fixed
Status: Resolved (was: Patch Available)
+1
Committed via GitHub. Thanks for the contribution [~adoroszlai] and thanks for
the review [~xyao].
> Refactor common test utilities to hadoop-hdds/common
>
>
> Key: HDDS-2684
> URL: https://issues.apache.org/jira/browse/HDDS-2684
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Expose test code from {{hadoop-hdds/common}} to other modules. Move some
> "common" test utilities. Example: random {{DatanodeDetails}} creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 merged pull request #322: HDDS-2684. Refactor common test utilities to hadoop-hdds/common
arp7 merged pull request #322: HDDS-2684. Refactor common test utilities to hadoop-hdds/common URL: https://github.com/apache/hadoop-ozone/pull/322 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 commented on issue #322: HDDS-2684. Refactor common test utilities to hadoop-hdds/common
arp7 commented on issue #322: HDDS-2684. Refactor common test utilities to hadoop-hdds/common URL: https://github.com/apache/hadoop-ozone/pull/322#issuecomment-565259614 +1 Looks like Xiaoyu's comments are addressed. Xiaoyu - if not we can address in a followup Jira. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #351: HDDS-1989. Fix ApplyTransaction error handling in OzoneManagerStateMachine.
bharatviswa504 opened a new pull request #351: HDDS-1989. Fix ApplyTransaction error handling in OzoneManagerStateMachine. URL: https://github.com/apache/hadoop-ozone/pull/351 ## What changes were proposed in this pull request? Terminate OM, after a critical error in apply transaction. Right now terminate for INTERNAL_ERROR and METADATA_ERROR. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-1989 Please replace this section with the link to the Apache JIRA) ## How was this patch tested? Need to see if any test for this failure scenario can be added. Mostly will try to come up with UT once HDDS-2682 merges in. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1989) Fix ApplyTransaction error handling in OzoneManagerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1989: - Labels: pull-request-available (was: ) > Fix ApplyTransaction error handling in OzoneManagerStateMachine > --- > > Key: HDDS-1989 > URL: https://issues.apache.org/jira/browse/HDDS-1989 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > Right now, applyTransaction calls validateAndUpdateCache. This does not > return any error. We should check the OmResponse Status code, and see if it > is critical error we should completeExceptionally and stop ratis server. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 closed pull request #350: WIP: HDDS-1989. Fix ApplyTransaction error handling in OzoneManagerStateMachine
bharatviswa504 closed pull request #350: WIP: HDDS-1989. Fix ApplyTransaction error handling in OzoneManagerStateMachine URL: https://github.com/apache/hadoop-ozone/pull/350 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #350: WIP: HDDS-1989. Fix ApplyTransaction error handling in OzoneManagerStateMachine
bharatviswa504 opened a new pull request #350: WIP: HDDS-1989. Fix ApplyTransaction error handling in OzoneManagerStateMachine URL: https://github.com/apache/hadoop-ozone/pull/350 ## What changes were proposed in this pull request? Right now, applyTransaction calls validateAndUpdateCache. This does not return any error. We should check the OmResponse Status code, and see if it is critical error we should stop OM. https://issues.apache.org/jira/browse/HDDS-1989 ## How was this patch tested? Existing HA tests ran, need to add any UT for this scenario. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2188) Implement LocatedFileStatus & getFileBlockLocations to provide node/localization information to Yarn/Mapreduce
[
https://issues.apache.org/jira/browse/HDDS-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-2188:
Status: Patch Available (was: In Progress)
> Implement LocatedFileStatus & getFileBlockLocations to provide
> node/localization information to Yarn/Mapreduce
> --
>
> Key: HDDS-2188
> URL: https://issues.apache.org/jira/browse/HDDS-2188
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Filesystem
>Affects Versions: 0.5.0
>Reporter: Mukul Kumar Singh
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> For applications like Hive/MapReduce to take advantage of the data locality
> in Ozone, Ozone should return the location of the Ozone blocks. This is
> needed for better read performance for Hadoop Applications.
> {code}
> if (file instanceof LocatedFileStatus) {
> blkLocations = ((LocatedFileStatus) file).getBlockLocations();
> } else {
> blkLocations = fs.getFileBlockLocations(file, 0, length);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2714) Add Ratis leader info for scm pipeline info command
[ https://issues.apache.org/jira/browse/HDDS-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle reassigned HDDS-2714: - Assignee: Siddharth Wagle > Add Ratis leader info for scm pipeline info command > --- > > Key: HDDS-2714 > URL: https://issues.apache.org/jira/browse/HDDS-2714 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.5.0 >Reporter: Shashikant Banerjee >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > > SCMCli prints the datanode info with the pipeline ID. it would be of great > help in debugging if it also displays the ratis leader info along with it as > this info is already populated by datanode to scm. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2716) ReadStateMachine failure should close down the ratis pipeline
[ https://issues.apache.org/jira/browse/HDDS-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle reassigned HDDS-2716: - Assignee: Siddharth Wagle > ReadStateMachine failure should close down the ratis pipeline > -- > > Key: HDDS-2716 > URL: https://issues.apache.org/jira/browse/HDDS-2716 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.5.0 >Reporter: Shashikant Banerjee >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > > Currently, in case of readStateMachineFailures, the containerStateMachine is > marked unhealthy which stalls all the pipeline operations. But the pipeline > will get destroyed in the next snapshot execution. The idea here is to just > destroy the pipeline as soon as readStateMachine call fails. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2703) OzoneFSInputStream to support ByteBufferReadable
[
https://issues.apache.org/jira/browse/HDDS-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995169#comment-16995169
]
YiSheng Lien commented on HDDS-2703:
Note:
Cause the current dependency version of Hadoop in Ozone is 3.2.0,
I don't implement {{ByteBufferPositionedReadable}} for now. (Supported in
Hadoop 3.3.0)
The same is {{StreamCapabilities}}, the {{StreamCapabilities}} support
{{StreamCapabilities.READBYTEBUFFER}} for Hadoop 3.3.0 now.
Should we create a JIRA for these ?
> OzoneFSInputStream to support ByteBufferReadable
>
>
> Key: HDDS-2703
> URL: https://issues.apache.org/jira/browse/HDDS-2703
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: YiSheng Lien
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This was found by [~cxorm] via HDDS-2443.
> ByteBufferReadable could help certain application performance, such as
> Impala. (See HDFS-14111)
> Additionally, if we support ByteBufferPositionedReadable, it would benefit
> HBase. (see HDFS-3246)
> Finally, we should add StreamCapabilities to let client probe for these
> abilities. (See HDFS-11644)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-2703) OzoneFSInputStream to support ByteBufferReadable
[
https://issues.apache.org/jira/browse/HDDS-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995169#comment-16995169
]
YiSheng Lien edited comment on HDDS-2703 at 12/12/19 10:41 PM:
---
Note:
Cause the current dependency version of Hadoop in Ozone is 3.2.0,
I don't implement {{ByteBufferPositionedReadable}} for now. (Supported in
Hadoop 3.3.0)
The same is {{StreamCapabilities}}, the {{StreamCapabilities}} support
{{StreamCapabilities.READBYTEBUFFER}} only for Hadoop 3.3.0 now.
Should we create a JIRA for these ?
was (Author: cxorm):
Note:
Cause the current dependency version of Hadoop in Ozone is 3.2.0,
I don't implement {{ByteBufferPositionedReadable}} for now. (Supported in
Hadoop 3.3.0)
The same is {{StreamCapabilities}}, the {{StreamCapabilities}} support
{{StreamCapabilities.READBYTEBUFFER}} for Hadoop 3.3.0 now.
Should we create a JIRA for these ?
> OzoneFSInputStream to support ByteBufferReadable
>
>
> Key: HDDS-2703
> URL: https://issues.apache.org/jira/browse/HDDS-2703
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: YiSheng Lien
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This was found by [~cxorm] via HDDS-2443.
> ByteBufferReadable could help certain application performance, such as
> Impala. (See HDFS-14111)
> Additionally, if we support ByteBufferPositionedReadable, it would benefit
> HBase. (see HDFS-3246)
> Finally, we should add StreamCapabilities to let client probe for these
> abilities. (See HDFS-11644)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel commented on issue #309: HDDS-2593. DatanodeAdminMonitor should track under replicated containers and complete the admin workflow accordingly
sodonnel commented on issue #309: HDDS-2593. DatanodeAdminMonitor should track under replicated containers and complete the admin workflow accordingly URL: https://github.com/apache/hadoop-ozone/pull/309#issuecomment-565213144 That test is still failing event after merging master to the branch. I though these commits would have fixed it, but I was confusing them with something else: ``` Addendum HDDS-2646. Start acceptance tests only if at least one THREE pipeline is available (#282) HDDS-2646. Start acceptance tests only if at least one THREE pipeline is available (#282) ``` I think this is a problem with the MockNodeManager handing the new operational states on this branch. I will look at it further. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cxorm commented on a change in pull request #345: HDDS-2703. OzoneFSInputStream to support ByteBufferReadable
cxorm commented on a change in pull request #345: HDDS-2703. OzoneFSInputStream
to support ByteBufferReadable
URL: https://github.com/apache/hadoop-ozone/pull/345#discussion_r357394244
##
File path:
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/OzoneFSInputStream.java
##
@@ -76,4 +79,21 @@ public boolean seekToNewSource(long targetPos) throws
IOException {
public int available() throws IOException {
return inputStream.available();
}
+
+ /**
+ * @param buf the ByteBuffer to receive the results of the read operation.
+ * @return the number of bytes read, possibly zero.
Review comment:
Thanks @jojochuang for the review.
Fixed the description, and updated the implementation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cxorm commented on a change in pull request #345: HDDS-2703. OzoneFSInputStream to support ByteBufferReadable
cxorm commented on a change in pull request #345: HDDS-2703. OzoneFSInputStream
to support ByteBufferReadable
URL: https://github.com/apache/hadoop-ozone/pull/345#discussion_r357393478
##
File path:
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/OzoneFSInputStream.java
##
@@ -76,4 +79,21 @@ public boolean seekToNewSource(long targetPos) throws
IOException {
public int available() throws IOException {
return inputStream.available();
}
+
+ /**
+ * @param buf the ByteBuffer to receive the results of the read operation.
+ * @return the number of bytes read, possibly zero.
+ * @throws IOException if there is some error performing the read
+ */
+ @Override
+ public int read(ByteBuffer buf) throws IOException {
+
+int bufInitPos = buf.position();
+while (buf.hasRemaining() && (inputStream.available() > 0)) {
Review comment:
Thanks @bharatviswa504 for the review.
Fixed, and work well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cxorm commented on a change in pull request #345: HDDS-2703. OzoneFSInputStream to support ByteBufferReadable
cxorm commented on a change in pull request #345: HDDS-2703. OzoneFSInputStream
to support ByteBufferReadable
URL: https://github.com/apache/hadoop-ozone/pull/345#discussion_r357393104
##
File path:
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/OzoneFSInputStream.java
##
@@ -76,4 +81,39 @@ public boolean seekToNewSource(long targetPos) throws
IOException {
public int available() throws IOException {
return inputStream.available();
}
+
+ /**
+ * @param buf the ByteBuffer to receive the results of the read operation.
+ * @return the number of bytes read, possibly zero, or -1 if
+ * reach end-of-stream
+ * @throws IOException if there is some error performing the read
+ */
+ @Override
+ public int read(ByteBuffer buf) throws IOException {
+
+int bufInitPos = buf.position();
+int readLen = Math.min(buf.remaining(), inputStream.available());
+
+byte[] readData = new byte[readLen];
+int bytesRead = inputStream.read(readData, bufInitPos, readLen);
+buf.put(readData);
+
+return bytesRead;
+ }
+
+ /**
+ * Query the stream for a specific capability.
+ *
+ * @param capability string to query the stream support for.
+ * @return True if the stream supports capability.
+ */
+ @Override
+ public boolean hasCapability(String capability) {
+switch (StringUtils.toLowerCase(capability)) {
+case "in:readbytebuffer":
Review comment:
This case should be ```StreamCapabilities.READBYTEBUFFER``` that is
supported in Hadoop 3.3.0 for now.
I use ```in:readbytebuffer``` here cause the dependency version of hadoop in
Ozone is 3.2.0.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] jojochuang commented on a change in pull request #345: HDDS-2703. OzoneFSInputStream to support ByteBufferReadable
jojochuang commented on a change in pull request #345: HDDS-2703.
OzoneFSInputStream to support ByteBufferReadable
URL: https://github.com/apache/hadoop-ozone/pull/345#discussion_r357386990
##
File path:
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/OzoneFSInputStream.java
##
@@ -76,4 +79,21 @@ public boolean seekToNewSource(long targetPos) throws
IOException {
public int available() throws IOException {
return inputStream.available();
}
+
+ /**
+ * @param buf the ByteBuffer to receive the results of the read operation.
+ * @return the number of bytes read, possibly zero.
Review comment:
According to the ByteBufferReadable javadoc, it should return -1 if end of
file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2443) Python client/interface for Ozone
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16995138#comment-16995138 ] Wei-Chiu Chuang commented on HDDS-2443: --- Thanks for the doc. It looks so easy to support Python clients! Could you make the instruction under Ozone Recipes? https://hadoop.apache.org/ozone/docs/0.4.1-alpha/recipe.html > Python client/interface for Ozone > - > > Key: HDDS-2443 > URL: https://issues.apache.org/jira/browse/HDDS-2443 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Client >Reporter: Li Cheng >Priority: Major > Attachments: Ozone with pyarrow.html, Ozone with pyarrow.odt, > OzoneS3.py > > > Original ideas: item#25 in > [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors] > Ozone Client(Python) for Data Science Notebook such as Jupyter. > # Size: Large > # PyArrow: [https://pypi.org/project/pyarrow/] > # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API > Impala uses libhdfs > > Path to try: > # s3 interface: Ozone s3 gateway(already supported) + AWS python client > (boto3) > # python native RPC > # pyarrow + libhdfs, which use the Java client under the hood. > # python + C interface of go / rust ozone library. I created POC go / rust > clients earlier which can be improved if the libhdfs interface is not good > enough. [By [~elek]] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2723) Wrap filesystem.open() with try-catch in TestOzoneFSInputStream
[ https://issues.apache.org/jira/browse/HDDS-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien updated HDDS-2723: --- Description: Thanks [~weichiu] for the proposal. To ensure resource is closed properly, we could wrap the fs.open() in a try. The detail is on https://github.com/apache/hadoop-ozone/pull/345. was: Thanks [~weichiu] for the proposal. The detail is on https://github.com/apache/hadoop-ozone/pull/345. > Wrap filesystem.open() with try-catch in TestOzoneFSInputStream > --- > > Key: HDDS-2723 > URL: https://issues.apache.org/jira/browse/HDDS-2723 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Filesystem >Reporter: YiSheng Lien >Assignee: YiSheng Lien >Priority: Major > > Thanks [~weichiu] for the proposal. > To ensure resource is closed properly, we could wrap the fs.open() in a try. > The detail is on https://github.com/apache/hadoop-ozone/pull/345. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2723) Wrap filesystem.open() with try-catch in TestOzoneFSInputStream
[ https://issues.apache.org/jira/browse/HDDS-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien updated HDDS-2723: --- Summary: Wrap filesystem.open() with try-catch in TestOzoneFSInputStream (was: Wrap filesystem.open() with try-catch to ensure resource is closed properly) > Wrap filesystem.open() with try-catch in TestOzoneFSInputStream > --- > > Key: HDDS-2723 > URL: https://issues.apache.org/jira/browse/HDDS-2723 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Filesystem >Reporter: YiSheng Lien >Assignee: YiSheng Lien >Priority: Major > > Thanks [~weichiu] for the proposal. > The detail is on https://github.com/apache/hadoop-ozone/pull/345. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cxorm commented on a change in pull request #345: HDDS-2703. OzoneFSInputStream to support ByteBufferReadable
cxorm commented on a change in pull request #345: HDDS-2703. OzoneFSInputStream
to support ByteBufferReadable
URL: https://github.com/apache/hadoop-ozone/pull/345#discussion_r357364575
##
File path:
hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFSInputStream.java
##
@@ -134,4 +135,20 @@ public void testO3FSMultiByteRead() throws IOException {
Assert.assertTrue(Arrays.equals(value, data));
inputStream.close();
}
+
+ @Test
+ public void testO3FSByteBufferRead() throws IOException {
+FSDataInputStream inputStream = fs.open(filePath);
Review comment:
Thanks @jojochuang for the suggest.
JIRA created for this issue.
([HDDS-2723](https://issues.apache.org/jira/browse/HDDS-2723))
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2723) Wrap filesystem.open() with try-catch to ensure resource is closed properly
[ https://issues.apache.org/jira/browse/HDDS-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien updated HDDS-2723: --- External issue ID: (was: HDDS-2703) > Wrap filesystem.open() with try-catch to ensure resource is closed properly > --- > > Key: HDDS-2723 > URL: https://issues.apache.org/jira/browse/HDDS-2723 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Filesystem > Environment: [PR of HDDS-2703|http://example.com] >Reporter: YiSheng Lien >Assignee: YiSheng Lien >Priority: Major > > Thanks [~weichiu] for the proposal. > The detail is on https://github.com/apache/hadoop-ozone/pull/345. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2723) Wrap filesystem.open() with try-catch to ensure resource is closed properly
[ https://issues.apache.org/jira/browse/HDDS-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien updated HDDS-2723: --- Environment: (was: [PR of HDDS-2703|http://example.com]) > Wrap filesystem.open() with try-catch to ensure resource is closed properly > --- > > Key: HDDS-2723 > URL: https://issues.apache.org/jira/browse/HDDS-2723 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Filesystem >Reporter: YiSheng Lien >Assignee: YiSheng Lien >Priority: Major > > Thanks [~weichiu] for the proposal. > The detail is on https://github.com/apache/hadoop-ozone/pull/345. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2723) Wrap filesystem.open() with try-catch to ensure resource is closed properly
YiSheng Lien created HDDS-2723: -- Summary: Wrap filesystem.open() with try-catch to ensure resource is closed properly Key: HDDS-2723 URL: https://issues.apache.org/jira/browse/HDDS-2723 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: Ozone Filesystem Environment: [PR of HDDS-2703|http://example.com] Reporter: YiSheng Lien Assignee: YiSheng Lien Thanks [~weichiu] for the proposal. The detail is on https://github.com/apache/hadoop-ozone/pull/345. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cxorm commented on a change in pull request #329: HDDS-2423. Add the recover-trash command client side code
cxorm commented on a change in pull request #329: HDDS-2423. Add the
recover-trash command client side code
URL: https://github.com/apache/hadoop-ozone/pull/329#discussion_r357352410
##
File path:
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
##
@@ -720,6 +720,14 @@ public void renameKey(String volumeName, String
bucketName,
keyPrefix, maxKeys);
}
+ @Override
+ public boolean recoverTrash(String volumeName, String bucketName,
+ String keyName, String destinationName) throws IOException {
Review comment:
Thanks @xiaoyuyao for this NIT.
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cxorm commented on a change in pull request #329: HDDS-2423. Add the recover-trash command client side code
cxorm commented on a change in pull request #329: HDDS-2423. Add the
recover-trash command client side code
URL: https://github.com/apache/hadoop-ozone/pull/329#discussion_r357350576
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -2238,6 +2238,15 @@ public void deleteKey(OmKeyArgs args) throws
IOException {
}
}
+ // TODO: HDDS-2424. recover-trash command server side handling.
+ @Override
+ public boolean recoverTrash(String volumeName, String bucketName,
+ String keyName, String destinationBucket) throws IOException {
+
+boolean recoverOperation = true;
+return recoverOperation;
Review comment:
Thanks @xiaoyuyao for the review.
I updated the UT that is prototype for now,
and it can be extended for the following server-side development.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] jojochuang commented on a change in pull request #345: HDDS-2703. OzoneFSInputStream to support ByteBufferReadable
jojochuang commented on a change in pull request #345: HDDS-2703.
OzoneFSInputStream to support ByteBufferReadable
URL: https://github.com/apache/hadoop-ozone/pull/345#discussion_r357315867
##
File path:
hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFSInputStream.java
##
@@ -134,4 +135,20 @@ public void testO3FSMultiByteRead() throws IOException {
Assert.assertTrue(Arrays.equals(value, data));
inputStream.close();
}
+
+ @Test
+ public void testO3FSByteBufferRead() throws IOException {
+FSDataInputStream inputStream = fs.open(filePath);
Review comment:
i would really love to see this wrapped in a try with block to ensure
resource is closed properly. But I see that all test cases in the file follows
the same pattern so maybe we can have a separate jira to correct that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2722) Let ChunkManager read/write ChunkBuffer instead of ByteBuffer
Attila Doroszlai created HDDS-2722:
--
Summary: Let ChunkManager read/write ChunkBuffer instead of
ByteBuffer
Key: HDDS-2722
URL: https://issues.apache.org/jira/browse/HDDS-2722
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: Ozone Datanode
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
Change {{ChunkManager}} read/write methods to accept/return {{ChunkBuffer}}
instead of {{ByteBuffer}}. This allows seamlessly passing multiple buffers
without further interface change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2721) Allow wrapping list of ByteBuffers with ChunkBuffer
Attila Doroszlai created HDDS-2721:
--
Summary: Allow wrapping list of ByteBuffers with ChunkBuffer
Key: HDDS-2721
URL: https://issues.apache.org/jira/browse/HDDS-2721
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
{{ChunkBuffer}} is a useful abstraction over {{ByteBuffer}} to hide whether
it's backed by a single {{ByteBuffer}} or multiple ones
({{IncrementalChunkBuffer}}). However, {{IncrementalChunkBuffer}} allocates
its own {{ByteBuffer}} instances and only works with uniform buffer sizes. The
goal of this task is to allow wrapping an existing {{List}} in
{{ChunkBuffer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2720) Ozone Failure injection Service
Prashant Pogde created HDDS-2720: Summary: Ozone Failure injection Service Key: HDDS-2720 URL: https://issues.apache.org/jira/browse/HDDS-2720 Project: Hadoop Distributed Data Store Issue Type: New Feature Components: Ozone Datanode, Ozone Filesystem, Ozone Manager, Ozone Recon, SCM Affects Versions: 0.5.0 Reporter: Prashant Pogde Assignee: Prashant Pogde Fix For: 0.5.0 Attachments: OzoneNoiseInjection.pdf This will be used to track development for failure injection service. This can be used to inject various failures/delays in an ozone cluster and validate ozone in presence of these failures or extreme conditions. Attached document provides a brief overview for this failure injection service and how it could be leveraged to validate ozone in stressful environments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2719) Rename S3Token to S3AuthInfo
[ https://issues.apache.org/jira/browse/HDDS-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2719: - Labels: pull-request-available (was: ) > Rename S3Token to S3AuthInfo > > > Key: HDDS-2719 > URL: https://issues.apache.org/jira/browse/HDDS-2719 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: S3 >Reporter: Mohammad >Assignee: Mohammad >Priority: Minor > Labels: pull-request-available > Fix For: 0.5.0 > > > Rename S3Token to something like S3AuthInfo > Item #24 in the list of ideas for new contributors to work on > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mohammadjkhan opened a new pull request #349: HDDS-2719. Rename S3Token to S3AuthInfo
mohammadjkhan opened a new pull request #349: HDDS-2719. Rename S3Token to S3AuthInfo URL: https://github.com/apache/hadoop-ozone/pull/349 ## What changes were proposed in this pull request? Renamed references of S3TOKEN to S3AUTHINFO ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2719 ## How was this patch tested? Ran project unit tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cxorm commented on issue #296: HDDS-2659. KeyValueHandler#handleCreateContainer should log the exception on container creation failure
cxorm commented on issue #296: HDDS-2659. KeyValueHandler#handleCreateContainer should log the exception on container creation failure URL: https://github.com/apache/hadoop-ozone/pull/296#issuecomment-565110871 Thanks @xiaoyuyao, @anuengineer and @bharatviswa504 for the reviews. And thanks @mukul1987 for the merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2035) Implement datanode level CLI to reveal pipeline info
[ https://issues.apache.org/jira/browse/HDDS-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2035: - Labels: pull-request-available (was: ) > Implement datanode level CLI to reveal pipeline info > > > Key: HDDS-2035 > URL: https://issues.apache.org/jira/browse/HDDS-2035 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Li Cheng >Priority: Major > Labels: pull-request-available > > 1. filter pipeline by datanode -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng opened a new pull request #348: HDDS-2035 Implement datanode level CLI to reveal pipeline relation.
timmylicheng opened a new pull request #348: HDDS-2035 Implement datanode level CLI to reveal pipeline relation. URL: https://github.com/apache/hadoop-ozone/pull/348 ## What changes were proposed in this pull request? #HDDS-2035 Implement datanode level CLI to reveal pipeline info ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2035 (Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HDDS-. Fix a typo in YYY.) Please replace this section with the link to the Apache JIRA) ## How was this patch tested? bash-4.2$ ozone scmcli datanode list Datanode: 7a68a668-3dbe-4366-9fed-271f221c1db0 (172.20.0.2/ozone_datanode_1.ozone_default). Related pipelines: cd060313-b724-46cf-a83a-8764ba066ae2/ONE/RATIS/OPEN (Please explain how this patch was tested. Ex: unit tests, manual tests) (If this patch involves UI changes, please attach a screen-shot; otherwise, remove this) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2675) Sonar : fix issues in PipelineStateManager
[ https://issues.apache.org/jira/browse/HDDS-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2675: - Labels: pull-request-available (was: ) > Sonar : fix issues in PipelineStateManager > -- > > Key: HDDS-2675 > URL: https://issues.apache.org/jira/browse/HDDS-2675 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Labels: pull-request-available > > Sonar "Code smell" category issues: > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-uCKcVY8lQ4ZsFf=AW5md-uCKcVY8lQ4ZsFf > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-uCKcVY8lQ4ZsFd=AW5md-uCKcVY8lQ4ZsFd > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-uCKcVY8lQ4ZsFg=AW5md-uCKcVY8lQ4ZsFg > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-uCKcVY8lQ4ZsFe=AW5md-uCKcVY8lQ4ZsFe > https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-uCKcVY8lQ4ZsFh=AW5md-uCKcVY8lQ4ZsFh -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] supratimdeka opened a new pull request #347: HDDS-2675. Sonar : fix issues in PipelineStateManager
supratimdeka opened a new pull request #347: HDDS-2675. Sonar : fix issues in PipelineStateManager URL: https://github.com/apache/hadoop-ozone/pull/347 ## What changes were proposed in this pull request? Resolved all Sonar "code smell" category issues in PipelineStateManager.java, including removal of unused parameter 'conf' from the constructor. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2675 ## How was this patch tested? existing junits: TestRatisPipelineProvider.java TestPipelineStateManager.java This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel edited a comment on issue #309: HDDS-2593. DatanodeAdminMonitor should track under replicated containers and complete the admin workflow accordingly
sodonnel edited a comment on issue #309: HDDS-2593. DatanodeAdminMonitor should track under replicated containers and complete the admin workflow accordingly URL: https://github.com/apache/hadoop-ozone/pull/309#issuecomment-565064730 The unit test failure is still this one, unrelated to the changes I believe: ```[ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.211 s <<< FAILURE! - in org.apache.hadoop.hdds.scm.block.TestBlockManager [ERROR] testMultipleBlockAllocationWithClosedContainer(org.apache.hadoop.hdds.scm.block.TestBlockManager) Time elapsed: 2.248 s <<< ERROR! org.apache.hadoop.hdds.scm.pipeline.InsufficientDatanodesException: Cannot create pipeline of factor 3 using 2 nodes. at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.create(RatisPipelineProvider.java:153) at org.apache.hadoop.hdds.scm.pipeline.PipelineFactory.create(PipelineFactory.java:58) at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.createPipeline(SCMPipelineManager.java:155) at org.apache.hadoop.hdds.scm.block.TestBlockManager.testMultipleBlockAllocationWithClosedContainer(TestBlockManager.java:306) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) ``` It should be fixed when we next merge master into this branch, as I believe that test has been fixed on master now. I agree there is a chance we could move the calculation into replication manager and track the metrics of under-replicated blocks in the NodeManager, but for now I think we should move forward and commit this and once we have a working decommission solution we can look to optimise and refactor it further. If you are happy can you please go ahead and commit this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel edited a comment on issue #309: HDDS-2593. DatanodeAdminMonitor should track under replicated containers and complete the admin workflow accordingly
sodonnel edited a comment on issue #309: HDDS-2593. DatanodeAdminMonitor should track under replicated containers and complete the admin workflow accordingly URL: https://github.com/apache/hadoop-ozone/pull/309#issuecomment-565064730 The unit test failure is still this one, unrelated to the changes I believe: ```[ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.211 s <<< FAILURE! - in org.apache.hadoop.hdds.scm.block.TestBlockManager [ERROR] testMultipleBlockAllocationWithClosedContainer(org.apache.hadoop.hdds.scm.block.TestBlockManager) Time elapsed: 2.248 s <<< ERROR! org.apache.hadoop.hdds.scm.pipeline.InsufficientDatanodesException: Cannot create pipeline of factor 3 using 2 nodes. at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.create(RatisPipelineProvider.java:153) at org.apache.hadoop.hdds.scm.pipeline.PipelineFactory.create(PipelineFactory.java:58) at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.createPipeline(SCMPipelineManager.java:155) at org.apache.hadoop.hdds.scm.block.TestBlockManager.testMultipleBlockAllocationWithClosedContainer(TestBlockManager.java:306) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) ``` It should be fixed when we next merge master into this branch, as I believe that test has been fixed on master now. I agree there is a chance we could move the calculation into replication manager and track the metrics of under-replicated blocks in the NodeManager, but for now I think we should move forward and commit this and once we have a working decommission solution we can look to optimised and refactor it further. If you are happy can you please go ahead and commit this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel commented on issue #309: HDDS-2593. DatanodeAdminMonitor should track under replicated containers and complete the admin workflow accordingly
sodonnel commented on issue #309: HDDS-2593. DatanodeAdminMonitor should track under replicated containers and complete the admin workflow accordingly URL: https://github.com/apache/hadoop-ozone/pull/309#issuecomment-565064730 The unit test failure is still this one, unrelated to the changes I believe: ```[ERROR] Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.211 s <<< FAILURE! - in org.apache.hadoop.hdds.scm.block.TestBlockManager [ERROR] testMultipleBlockAllocationWithClosedContainer(org.apache.hadoop.hdds.scm.block.TestBlockManager) Time elapsed: 2.248 s <<< ERROR! org.apache.hadoop.hdds.scm.pipeline.InsufficientDatanodesException: Cannot create pipeline of factor 3 using 2 nodes. at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.create(RatisPipelineProvider.java:153) at org.apache.hadoop.hdds.scm.pipeline.PipelineFactory.create(PipelineFactory.java:58) at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.createPipeline(SCMPipelineManager.java:155) at org.apache.hadoop.hdds.scm.block.TestBlockManager.testMultipleBlockAllocationWithClosedContainer(TestBlockManager.java:306) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)``` It should be fixed when we next merge master into this branch, as I believe that test has been fixed on master now. I agree there is a chance we could move the calculation into replication manager and track the metrics of under-replicated blocks in the NodeManager, but for now I think we should move forward and commit this and once we have a working decommission solution we can look to optimised and refactor it further. If you are happy can you please go ahead and commit this change? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #337: HDDS-2706. Add class CRLCodec - used for certificate revocation list.
elek commented on issue #337: HDDS-2706. Add class CRLCodec - used for certificate revocation list. URL: https://github.com/apache/hadoop-ozone/pull/337#issuecomment-565018418 Thank you very much the patch @abhishekaypurohit Is there any related design doc which can help to understand the goal and review the change? I didn't find it in the Jira but I might miss it... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1792) Use ConcurrentHashSet in place of ConcurrentHashMap in ContainerStteMachine.
[ https://issues.apache.org/jira/browse/HDDS-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh resolved HDDS-1792. - Resolution: Invalid > Use ConcurrentHashSet in place of ConcurrentHashMap in ContainerStteMachine. > > > Key: HDDS-1792 > URL: https://issues.apache.org/jira/browse/HDDS-1792 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > > Use ConcurrentHashSet in place of ConcurrentHashMap in ContainerStteMachine. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1679) TestBCSID failing because of dangling db references
[ https://issues.apache.org/jira/browse/HDDS-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh resolved HDDS-1679. - Resolution: Cannot Reproduce Haven't seen this in a while. resolving this. > TestBCSID failing because of dangling db references > --- > > Key: HDDS-1679 > URL: https://issues.apache.org/jira/browse/HDDS-1679 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > TestBCSID failing because of dangling db references. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2663) Add OzoneFileSystem to MiniOzoneChaosCluster
[ https://issues.apache.org/jira/browse/HDDS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh resolved HDDS-2663. - Resolution: Duplicate > Add OzoneFileSystem to MiniOzoneChaosCluster > > > Key: HDDS-2663 > URL: https://issues.apache.org/jira/browse/HDDS-2663 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > > This jira adds ozonefilesystem functionality to MiniOzoneChaosCluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2719) Rename S3Token to S3AuthInfo
[ https://issues.apache.org/jira/browse/HDDS-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad updated HDDS-2719: --- Fix Version/s: (was: 0.4.2) 0.5.0 > Rename S3Token to S3AuthInfo > > > Key: HDDS-2719 > URL: https://issues.apache.org/jira/browse/HDDS-2719 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Mohammad >Assignee: Mohammad >Priority: Minor > Fix For: 0.5.0 > > > Rename S3Token to something like S3AuthInfo > Item #24 in the list of ideas for new contributors to work on > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2719) Rename S3Token to S3AuthInfo
[ https://issues.apache.org/jira/browse/HDDS-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad updated HDDS-2719: --- Component/s: S3 > Rename S3Token to S3AuthInfo > > > Key: HDDS-2719 > URL: https://issues.apache.org/jira/browse/HDDS-2719 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: S3 >Reporter: Mohammad >Assignee: Mohammad >Priority: Minor > Fix For: 0.5.0 > > > Rename S3Token to something like S3AuthInfo > Item #24 in the list of ideas for new contributors to work on > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2719) Rename S3Token to S3AuthInfo
[ https://issues.apache.org/jira/browse/HDDS-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad updated HDDS-2719: --- Component/s: (was: SCM) > Rename S3Token to S3AuthInfo > > > Key: HDDS-2719 > URL: https://issues.apache.org/jira/browse/HDDS-2719 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Mohammad >Assignee: Mohammad >Priority: Minor > Fix For: 0.4.2 > > > Rename S3Token to something like S3AuthInfo > Item #24 in the list of ideas for new contributors to work on > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2719) Rename S3Token to S3AuthInfo
[ https://issues.apache.org/jira/browse/HDDS-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohammad updated HDDS-2719: --- Description: Rename S3Token to something like S3AuthInfo Item #24 in the list of ideas for new contributors to work on https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors was: Sonar report : [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-_2KcVY8lQ4ZsVd=false=BUG|https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-4jKcVY8lQ4ZsPQ=AW5md-4jKcVY8lQ4ZsPQ] > Rename S3Token to S3AuthInfo > > > Key: HDDS-2719 > URL: https://issues.apache.org/jira/browse/HDDS-2719 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Mohammad >Assignee: Mohammad >Priority: Minor > Fix For: 0.4.2 > > > Rename S3Token to something like S3AuthInfo > Item #24 in the list of ideas for new contributors to work on > https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2719) Rename S3Token to S3AuthInfo
Mohammad created HDDS-2719: -- Summary: Rename S3Token to S3AuthInfo Key: HDDS-2719 URL: https://issues.apache.org/jira/browse/HDDS-2719 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Reporter: Mohammad Assignee: Mohammad Fix For: 0.4.2 Sonar report : [https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-_2KcVY8lQ4ZsVd=false=BUG|https://sonarcloud.io/project/issues?id=hadoop-ozone=AW5md-4jKcVY8lQ4ZsPQ=AW5md-4jKcVY8lQ4ZsPQ] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2588) Consolidate compose environments
[
https://issues.apache.org/jira/browse/HDDS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-2588:
---
Issue Type: Improvement (was: Bug)
> Consolidate compose environments
>
>
> Key: HDDS-2588
> URL: https://issues.apache.org/jira/browse/HDDS-2588
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: docker
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are a few slightly different sample docker compose environments: ozone,
> ozoneperf, ozones3, ozone-recon. This issue proposes to merge these 4 by
> minor additions to ozoneperf:
> # add {{recon}} service from {{ozone-recon}}
> # run GDPR and S3 tests
> # expose datanode web port (eg. for profiling)
> Plus: also run ozone-shell test (from basic suite), which is currently run
> only in ozonesecure
> I also propose to rename {{ozoneperf}} to {{ozone}} for simplicity.
> Consolidating these 4 environments would slightly reduce both code
> duplication and the time needed for acceptance tests.
> CC [~elek]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2588) Consolidate compose environments
[
https://issues.apache.org/jira/browse/HDDS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-2588:
---
Fix Version/s: 0.5.0
> Consolidate compose environments
>
>
> Key: HDDS-2588
> URL: https://issues.apache.org/jira/browse/HDDS-2588
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: docker
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are a few slightly different sample docker compose environments: ozone,
> ozoneperf, ozones3, ozone-recon. This issue proposes to merge these 4 by
> minor additions to ozoneperf:
> # add {{recon}} service from {{ozone-recon}}
> # run GDPR and S3 tests
> # expose datanode web port (eg. for profiling)
> Plus: also run ozone-shell test (from basic suite), which is currently run
> only in ozonesecure
> I also propose to rename {{ozoneperf}} to {{ozone}} for simplicity.
> Consolidating these 4 environments would slightly reduce both code
> duplication and the time needed for acceptance tests.
> CC [~elek]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #238: HDDS-2588. Consolidate compose environments
adoroszlai commented on issue #238: HDDS-2588. Consolidate compose environments URL: https://github.com/apache/hadoop-ozone/pull/238#issuecomment-564959372 Thanks @anuengineer, @avijayanhwx, @bharatviswa504 for review, and @elek for review and commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #324: HDDS-2688. Refactor container response builders to hadoop-hdds/common
adoroszlai commented on issue #324: HDDS-2688. Refactor container response builders to hadoop-hdds/common URL: https://github.com/apache/hadoop-ozone/pull/324#issuecomment-564958466 Thanks @bshashikant for reviewing and committing this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2659) KeyValueHandler#handleCreateContainer should log the exception on container creation failure
[
https://issues.apache.org/jira/browse/HDDS-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh resolved HDDS-2659.
-
Fix Version/s: 0.5.0
Resolution: Fixed
Thanks for the contribution [~cxorm]. I have committed this.
> KeyValueHandler#handleCreateContainer should log the exception on container
> creation failure
>
>
> Key: HDDS-2659
> URL: https://issues.apache.org/jira/browse/HDDS-2659
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nilotpal Nandi
>Assignee: YiSheng Lien
>Priority: Major
> Labels: newbie, pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> KeyValueHandler#handleCreateContainer does not log the stack trace of the
> StorageContainerException cause. This leads to difficulty in debugging the
> reason for failure.
>
> {code:java}
> 2019-11-04 00:40:56,719 INFO
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Operation:
> CreateContainer : Trace ID:
> 1f7530ed73cf2e47:74588d4de9f1d702:1f7530ed73cf2e47:0 : Message: Container
> creation failed. : Result: CONTAINER_INTERNAL_ERROR
> 2019-11-04 00:40:56,720 INFO
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher: Operation:
> WriteChunk : Trace ID: 1f7530ed73cf2e47:74588d4de9f1d702:1f7530ed73cf2e47:0 :
> Message: ContainerID 11 creation failed : Result: CONTAINER_INTERNAL_ERROR
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek closed pull request #238: HDDS-2588. Consolidate compose environments
elek closed pull request #238: HDDS-2588. Consolidate compose environments URL: https://github.com/apache/hadoop-ozone/pull/238 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 merged pull request #296: HDDS-2659. KeyValueHandler#handleCreateContainer should log the exception on container creation failure
mukul1987 merged pull request #296: HDDS-2659. KeyValueHandler#handleCreateContainer should log the exception on container creation failure URL: https://github.com/apache/hadoop-ozone/pull/296 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2688) Refactor container response builders to hdds-common
[
https://issues.apache.org/jira/browse/HDDS-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDDS-2688:
--
Fix Version/s: 0.5.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks [~adoroszlai] for working on this. I have committed this.
> Refactor container response builders to hdds-common
> ---
>
> Key: HDDS-2688
> URL: https://issues.apache.org/jira/browse/HDDS-2688
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{ContainerUtils}} and {{BlockUtils}} have some helper functions to build
> responses to container commands. These would be useful for client-side unit
> tests, but {{client}} does not depend on {{container-service}} since the
> interfaces and messages it needs are defined in {{common}}. This issue
> proposes to move these helpers to {{common}} to avoid duplicating the
> functionality for tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant merged pull request #324: HDDS-2688. Refactor container response builders to hadoop-hdds/common
bshashikant merged pull request #324: HDDS-2688. Refactor container response builders to hadoop-hdds/common URL: https://github.com/apache/hadoop-ozone/pull/324 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2718) Send chunks in increments to datanode
Attila Doroszlai created HDDS-2718: -- Summary: Send chunks in increments to datanode Key: HDDS-2718 URL: https://issues.apache.org/jira/browse/HDDS-2718 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Client Reporter: Attila Doroszlai Assignee: Attila Doroszlai Change client to send chunks in multiple parts based on stream buffer increment size config. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2717) Handle chunk increments in datanode
[ https://issues.apache.org/jira/browse/HDDS-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-2717: --- Issue Type: Improvement (was: Task) > Handle chunk increments in datanode > --- > > Key: HDDS-2717 > URL: https://issues.apache.org/jira/browse/HDDS-2717 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Major > > Let datanode handle incremental additions to chunks (data with non-zero > offset). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2717) Handle chunk increments in datanode
Attila Doroszlai created HDDS-2717: -- Summary: Handle chunk increments in datanode Key: HDDS-2717 URL: https://issues.apache.org/jira/browse/HDDS-2717 Project: Hadoop Distributed Data Store Issue Type: Task Components: Ozone Datanode Reporter: Attila Doroszlai Assignee: Attila Doroszlai Let datanode handle incremental additions to chunks (data with non-zero offset). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2716) ReadStateMachine failure should close down the ratis pipeline
Shashikant Banerjee created HDDS-2716: - Summary: ReadStateMachine failure should close down the ratis pipeline Key: HDDS-2716 URL: https://issues.apache.org/jira/browse/HDDS-2716 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Affects Versions: 0.5.0 Reporter: Shashikant Banerjee Fix For: 0.5.0 Currently, in case of readStateMachineFailures, the containerStateMachine is marked unhealthy which stalls all the pipeline operations. But the pipeline will get destroyed in the next snapshot execution. The idea here is to just destroy the pipeline as soon as readStateMachine call fails. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #323: HDDS-2686. Use protobuf 3 instead of protobuf 2
elek commented on issue #323: HDDS-2686. Use protobuf 3 instead of protobuf 2
URL: https://github.com/apache/hadoop-ozone/pull/323#issuecomment-564928421
> but there are acceptance test failures.
Yes, the problem is in the mr test with hadoop-32. With hadoop 2.7 - 3.1 we
have no problems as we use properly isolated classpath environments ("legacy"
ozonesf jar) but until now we were compatible with hadoop 3.2.
With this change the `current` jar will be compatible with the (unreleased)
hadoop 3.3 and we need to use the classpath separation for 3.2.
I updated the tests and created a new mr test for 3.3 (based on a custom
docker image which can be replaced after the release with the official apache
image).
Note: shading is not working for the protobuf only the classpath isolation.
With the `current` jar we are not shading the hadoop API which includes
references to the protobuf GenericMessage. Without shading the Hadoop API we
can't shade the protobuf files, therefore the only option is to use the
classpath isolation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2715) Add config for client buffer increment size
Attila Doroszlai created HDDS-2715:
--
Summary: Add config for client buffer increment size
Key: HDDS-2715
URL: https://issues.apache.org/jira/browse/HDDS-2715
Project: Hadoop Distributed Data Store
Issue Type: Task
Components: Ozone Client
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
Introduce a new config property for client-side buffer increment size along the
lines of {{ozone.client.stream.buffer.flush.size}} and
{{ozone.client.stream.buffer.max.size}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2714) Add Ratis leader info for scm pipeline info command
Shashikant Banerjee created HDDS-2714: - Summary: Add Ratis leader info for scm pipeline info command Key: HDDS-2714 URL: https://issues.apache.org/jira/browse/HDDS-2714 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Affects Versions: 0.5.0 Reporter: Shashikant Banerjee Fix For: 0.5.0 SCMCli prints the datanode info with the pipeline ID. it would be of great help in debugging if it also displays the ratis leader info along with it as this info is already populated by datanode to scm. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2713) Decouple client write size and datanode chunk size
Attila Doroszlai created HDDS-2713: -- Summary: Decouple client write size and datanode chunk size Key: HDDS-2713 URL: https://issues.apache.org/jira/browse/HDDS-2713 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: Ozone Datanode, SCM Client Reporter: Attila Doroszlai Assignee: Attila Doroszlai Currently Datanode creates chunk files as they are received from HDDS client. This creates a conflict between requirements: client would like to use less memory for buffering, but datanode needs to avoid small files. The goal of this task is to decouple client and server write sizes to allow the client to send data in smaller increments without affecting datanode storage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
