[jira] [Resolved] (HDDS-2497) SafeMode check should allow key creation on single node pipeline when replication factor is 1
[
https://issues.apache.org/jira/browse/HDDS-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen resolved HDDS-2497.
--
Resolution: Resolved
It's resolved by HDDS-2034.
> SafeMode check should allow key creation on single node pipeline when
> replication factor is 1
> -
>
> Key: HDDS-2497
> URL: https://issues.apache.org/jira/browse/HDDS-2497
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Sammi Chen
>Priority: Major
>
> Start a single datanode ozone docker-compose with replication factor of 1.
> {code:java}
> OZONE-SITE.XML_ozone.replication=1{code}
> The key creation failed with Safemode exception below.
> {code:java}
> >$ docker-compose exec om bash
> bash-4.2$ ozone sh vol create /vol1
> bash-4.2$ ozone sh bucket create /vol1/bucket1
> ozone sh kbash-4.2$ ozone sh key put /vol1/bucket1/key1 README.md
> SCM_IN_SAFE_MODE SafeModePrecheck failed for allocateBlock{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] iamabug commented on a change in pull request #339: HDDS-2553. Iterator.next() methods should throw NoSuchElementException
iamabug commented on a change in pull request #339: HDDS-2553. Iterator.next()
methods should throw NoSuchElementException
URL: https://github.com/apache/hadoop-ozone/pull/339#discussion_r356443428
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChunkBufferImplWithByteBuffer.java
##
@@ -53,6 +54,9 @@ public boolean hasNext() {
@Override
public ByteBuffer next() {
final ByteBuffer duplicated = buffer.duplicate();
+if (!buffer.hasRemaining()) {
Review comment:
@xiaoyuyao Please review my unit test
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2177) Add a srubber thread to detect creation failure pipelines in ALLOCATED state
[ https://issues.apache.org/jira/browse/HDDS-2177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen resolved HDDS-2177. -- Resolution: Duplicate > Add a srubber thread to detect creation failure pipelines in ALLOCATED state > > > Key: HDDS-2177 > URL: https://issues.apache.org/jira/browse/HDDS-2177 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2099) Refactor to create pipeline via DN heartbeat response
[ https://issues.apache.org/jira/browse/HDDS-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen resolved HDDS-2099. -- Resolution: Duplicate > Refactor to create pipeline via DN heartbeat response > - > > Key: HDDS-2099 > URL: https://issues.apache.org/jira/browse/HDDS-2099 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > > Currently, SCM directly talk to DN GRPC server to create pipeline in a > background thread. We should avoid direct communication from SCM to DN for > better scalability of ozone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2099) Refactor to create pipeline via DN heartbeat response
[ https://issues.apache.org/jira/browse/HDDS-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen reassigned HDDS-2099: Assignee: Sammi Chen > Refactor to create pipeline via DN heartbeat response > - > > Key: HDDS-2099 > URL: https://issues.apache.org/jira/browse/HDDS-2099 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > > Currently, SCM directly talk to DN GRPC server to create pipeline in a > background thread. We should avoid direct communication from SCM to DN for > better scalability of ozone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
timmylicheng commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356431963
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -170,12 +169,18 @@ public Pipeline create(ReplicationFactor factor) throws
IOException {
throw new IllegalStateException("Unknown factor: " + factor.name());
}
+int nodeIdHash = 0;
+if (factor == ReplicationFactor.THREE) {
Review comment:
Good catch. Updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
timmylicheng commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356431684
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -100,4 +105,45 @@ static void destroyPipeline(DatanodeDetails dn,
PipelineID pipelineID,
true, p.getId());
}
}
+
+ static int encodeNodeIdsOfFactorThreePipeline(List nodes) {
+if (nodes.size() != HddsProtos.ReplicationFactor.THREE.getNumber()) {
+ return 0;
+}
+List nodeIds = nodes.stream()
+.map(DatanodeDetails::getUuid).distinct()
+.collect(Collectors.toList());
+nodeIds.sort(new ComparableComparator());
Review comment:
I will use xor then
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #333: HDDS-2279. Ozone S3 CLI commands not working on HA cluster
bharatviswa504 commented on a change in pull request #333: HDDS-2279. Ozone S3
CLI commands not working on HA cluster
URL: https://github.com/apache/hadoop-ozone/pull/333#discussion_r356421714
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/OzoneAddress.java
##
@@ -124,6 +126,41 @@ public OzoneClient createClient(OzoneConfiguration conf)
return client;
}
+ /**
+ * Create OzoneClient for S3Commands.
+ * @param conf
+ * @param omServiceID
+ * @return OzoneClient
+ * @throws IOException
+ * @throws OzoneClientException
+ */
+ public OzoneClient createClientForS3Commands(OzoneConfiguration conf,
+ String omServiceID)
+ throws IOException, OzoneClientException {
+if (omServiceID != null) {
+ // OM HA cluster
+ if (OmUtils.isOmHAServiceId(conf, omServiceID)) {
+return OzoneClientFactory.getRpcClient(omServiceID, conf);
+ } else {
+throw new OzoneClientException("Service ID specified does not match" +
+" with " + OZONE_OM_SERVICE_IDS_KEY + " defined in the " +
+"configuration. Configured " + OZONE_OM_SERVICE_IDS_KEY + " are" +
+conf.getTrimmedStringCollection(OZONE_OM_SERVICE_IDS_KEY));
+ }
+} else {
+ // If om service id is not specified, consider it as a non-HA cluster.
Review comment:
And also it can have multiple serviceID's when it has conf of local HA
cluster and remote HA cluster.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #333: HDDS-2279. Ozone S3 CLI commands not working on HA cluster
bharatviswa504 commented on a change in pull request #333: HDDS-2279. Ozone S3
CLI commands not working on HA cluster
URL: https://github.com/apache/hadoop-ozone/pull/333#discussion_r356420818
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/OzoneAddress.java
##
@@ -124,6 +126,41 @@ public OzoneClient createClient(OzoneConfiguration conf)
return client;
}
+ /**
+ * Create OzoneClient for S3Commands.
+ * @param conf
+ * @param omServiceID
+ * @return OzoneClient
+ * @throws IOException
+ * @throws OzoneClientException
+ */
+ public OzoneClient createClientForS3Commands(OzoneConfiguration conf,
+ String omServiceID)
+ throws IOException, OzoneClientException {
+if (omServiceID != null) {
+ // OM HA cluster
+ if (OmUtils.isOmHAServiceId(conf, omServiceID)) {
+return OzoneClientFactory.getRpcClient(omServiceID, conf);
+ } else {
+throw new OzoneClientException("Service ID specified does not match" +
+" with " + OZONE_OM_SERVICE_IDS_KEY + " defined in the " +
+"configuration. Configured " + OZONE_OM_SERVICE_IDS_KEY + " are" +
+conf.getTrimmedStringCollection(OZONE_OM_SERVICE_IDS_KEY));
+ }
+} else {
+ // If om service id is not specified, consider it as a non-HA cluster.
Review comment:
I too thought that, but later I got a doubt that if conf has service ID, but
it is of the remote cluster. (I think this can happen when conf has remote HA
cluster details and the local non-HA cluster) . So, thrown exception and in
this way passing service id is mandatory for HA cluster. Let me know your view
on this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #333: HDDS-2279. Ozone S3 CLI commands not working on HA cluster
bharatviswa504 commented on a change in pull request #333: HDDS-2279. Ozone S3
CLI commands not working on HA cluster
URL: https://github.com/apache/hadoop-ozone/pull/333#discussion_r356420818
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/OzoneAddress.java
##
@@ -124,6 +126,41 @@ public OzoneClient createClient(OzoneConfiguration conf)
return client;
}
+ /**
+ * Create OzoneClient for S3Commands.
+ * @param conf
+ * @param omServiceID
+ * @return OzoneClient
+ * @throws IOException
+ * @throws OzoneClientException
+ */
+ public OzoneClient createClientForS3Commands(OzoneConfiguration conf,
+ String omServiceID)
+ throws IOException, OzoneClientException {
+if (omServiceID != null) {
+ // OM HA cluster
+ if (OmUtils.isOmHAServiceId(conf, omServiceID)) {
+return OzoneClientFactory.getRpcClient(omServiceID, conf);
+ } else {
+throw new OzoneClientException("Service ID specified does not match" +
+" with " + OZONE_OM_SERVICE_IDS_KEY + " defined in the " +
+"configuration. Configured " + OZONE_OM_SERVICE_IDS_KEY + " are" +
+conf.getTrimmedStringCollection(OZONE_OM_SERVICE_IDS_KEY));
+ }
+} else {
+ // If om service id is not specified, consider it as a non-HA cluster.
Review comment:
I too thought that, but later I got a doubt that if conf has service ID, but
it is of the remote cluster. (I think this can happen when conf has remote HA
cluster details and the local non-HA cluster) . So, thrown exception and have
marked this mandatory for HA cluster. Let me know your view on this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData.
lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData. URL: https://github.com/apache/hadoop-ozone/pull/310#issuecomment-564398759 @adoroszlai I have created https://issues.apache.org/jira/browse/RATIS-776 which addresses the issue you pointed out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #331: HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro…
bharatviswa504 commented on a change in pull request #331: HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro… URL: https://github.com/apache/hadoop-ozone/pull/331#discussion_r356419503 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java ## @@ -1077,6 +1077,7 @@ private GetFileStatusResponse getOzoneFileStatus( .setVolumeName(keyArgs.getVolumeName()) .setBucketName(keyArgs.getBucketName()) .setKeyName(keyArgs.getKeyName()) +.setRefreshPipeline(keyArgs.getRefreshPipeline()) Review comment: So is that missing in OzoneManagerRequestHandler.java? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] iamabug commented on a change in pull request #339: HDDS-2553. Iterator.next() methods should throw NoSuchElementException
iamabug commented on a change in pull request #339: HDDS-2553. Iterator.next()
methods should throw NoSuchElementException
URL: https://github.com/apache/hadoop-ozone/pull/339#discussion_r356415532
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChunkBufferImplWithByteBuffer.java
##
@@ -53,6 +54,9 @@ public boolean hasNext() {
@Override
public ByteBuffer next() {
final ByteBuffer duplicated = buffer.duplicate();
+if (!buffer.hasRemaining()) {
Review comment:
And is rebase required here ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] iamabug commented on a change in pull request #339: HDDS-2553. Iterator.next() methods should throw NoSuchElementException
iamabug commented on a change in pull request #339: HDDS-2553. Iterator.next()
methods should throw NoSuchElementException
URL: https://github.com/apache/hadoop-ozone/pull/339#discussion_r356415267
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChunkBufferImplWithByteBuffer.java
##
@@ -53,6 +54,9 @@ public boolean hasNext() {
@Override
public ByteBuffer next() {
final ByteBuffer duplicated = buffer.duplicate();
+if (!buffer.hasRemaining()) {
Review comment:
Moving to Line 56 is ok, but could you give some advice on uni test here ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #331: HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro…
avijayanhwx commented on a change in pull request #331: HDDS-2188. Implement
LocatedFileStatus & getFileBlockLocations to pro…
URL: https://github.com/apache/hadoop-ozone/pull/331#discussion_r356414137
##
File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
##
@@ -677,6 +677,7 @@ message KeyArgs {
// request type.
optional uint64 modificationTime = 13;
optional bool sortDatanodes = 14;
+optional bool refreshPipeline = 15;
Review comment:
Yes, I agree. I can make the change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #331: HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro…
avijayanhwx commented on a change in pull request #331: HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro… URL: https://github.com/apache/hadoop-ozone/pull/331#discussion_r356413364 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java ## @@ -1077,6 +1077,7 @@ private GetFileStatusResponse getOzoneFileStatus( .setVolumeName(keyArgs.getVolumeName()) .setBucketName(keyArgs.getBucketName()) .setKeyName(keyArgs.getKeyName()) +.setRefreshPipeline(keyArgs.getRefreshPipeline()) Review comment: @bharatviswa504 Yes, FS listStatus API is invoked by MapReduce jobs to get the located file blocks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #333: HDDS-2279. Ozone S3 CLI commands not working on HA cluster
xiaoyuyao commented on a change in pull request #333: HDDS-2279. Ozone S3 CLI
commands not working on HA cluster
URL: https://github.com/apache/hadoop-ozone/pull/333#discussion_r356412598
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/OzoneAddress.java
##
@@ -124,6 +126,41 @@ public OzoneClient createClient(OzoneConfiguration conf)
return client;
}
+ /**
+ * Create OzoneClient for S3Commands.
+ * @param conf
+ * @param omServiceID
+ * @return OzoneClient
+ * @throws IOException
+ * @throws OzoneClientException
+ */
+ public OzoneClient createClientForS3Commands(OzoneConfiguration conf,
+ String omServiceID)
+ throws IOException, OzoneClientException {
+if (omServiceID != null) {
+ // OM HA cluster
+ if (OmUtils.isOmHAServiceId(conf, omServiceID)) {
+return OzoneClientFactory.getRpcClient(omServiceID, conf);
+ } else {
+throw new OzoneClientException("Service ID specified does not match" +
+" with " + OZONE_OM_SERVICE_IDS_KEY + " defined in the " +
+"configuration. Configured " + OZONE_OM_SERVICE_IDS_KEY + " are" +
+conf.getTrimmedStringCollection(OZONE_OM_SERVICE_IDS_KEY));
+ }
+} else {
+ // If om service id is not specified, consider it as a non-HA cluster.
Review comment:
Can we handle the HA case where om service ID is not provided by CLI
argument but local conf has a valid service ID without the Exception?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #332: HDDS-2539. Sonar: Fix sonar warnings in SCMContainerManager class.
swagle commented on a change in pull request #332: HDDS-2539. Sonar: Fix sonar warnings in SCMContainerManager class. URL: https://github.com/apache/hadoop-ozone/pull/332#discussion_r356411308 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java ## @@ -79,19 +77,14 @@ * Constructs a mapping class that creates mapping between container names * and pipelines. * - * @param nodeManager - NodeManager so that we can get the nodes that are - * healthy to place new - * containers. * passed to LevelDB and this memory is allocated in Native code space. * CacheSize is specified * in MB. Review comment: Addressed @xiaoyuyao, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #329: HDDS-2423. Add the recover-trash command client side code
xiaoyuyao commented on a change in pull request #329: HDDS-2423. Add the
recover-trash command client side code
URL: https://github.com/apache/hadoop-ozone/pull/329#discussion_r356411104
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -2238,6 +2238,15 @@ public void deleteKey(OmKeyArgs args) throws
IOException {
}
}
+ // TODO: HDDS-2424. recover-trash command server side handling.
+ @Override
+ public boolean recoverTrash(String volumeName, String bucketName,
+ String keyName, String destinationBucket) throws IOException {
+
+boolean recoverOperation = true;
+return recoverOperation;
Review comment:
Even this is a partial work, is IT possible to add a unit test for it, which
can be easily extended later for HDDS-2424.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #329: HDDS-2423. Add the recover-trash command client side code
xiaoyuyao commented on a change in pull request #329: HDDS-2423. Add the
recover-trash command client side code
URL: https://github.com/apache/hadoop-ozone/pull/329#discussion_r356410831
##
File path:
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
##
@@ -720,6 +720,14 @@ public void renameKey(String volumeName, String
bucketName,
keyPrefix, maxKeys);
}
+ @Override
+ public boolean recoverTrash(String volumeName, String bucketName,
+ String keyName, String destinationName) throws IOException {
Review comment:
NIT: can we rename destinationName to destinationBucket to be consistent?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #313: HDDS-2242. Avoid unnecessary rpc needed to discover the pipeline leader.
swagle commented on a change in pull request #313: HDDS-2242. Avoid unnecessary
rpc needed to discover the pipeline leader.
URL: https://github.com/apache/hadoop-ozone/pull/313#discussion_r356410606
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestRatisPipelineLeader.java
##
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.scm;
+
+import java.util.List;
+import java.util.Optional;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import org.apache.hadoop.ozone.HddsDatanodeService;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.ratis.grpc.client.GrpcClientProtocolClient;
+import org.apache.ratis.protocol.ClientId;
+import org.apache.ratis.protocol.GroupInfoReply;
+import org.apache.ratis.protocol.GroupInfoRequest;
+import org.apache.ratis.protocol.RaftGroupId;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Test pipeline leader information is correctly used.
+ */
+public class TestRatisPipelineLeader {
+ private static MiniOzoneCluster cluster;
+ private static OzoneConfiguration conf;
+
+ @BeforeClass
+ public static void setup() throws Exception {
+conf = new OzoneConfiguration();
+cluster = MiniOzoneCluster
+.newBuilder(conf)
+.setNumDatanodes(3)
+.build();
+cluster.waitForClusterToBeReady();
+ }
+
+ @AfterClass
+ public static void shutdown() throws Exception {
+if (cluster != null) {
+ cluster.shutdown();
+}
+ }
+
+ @Test
+ public void testLeaderIdUsedOnFirstCall() throws Exception {
+List pipelines = cluster.getStorageContainerManager()
+.getPipelineManager().getPipelines();
+Optional ratisPipeline = pipelines.stream().filter(p ->
+p.getType().equals(HddsProtos.ReplicationType.RATIS)).findFirst();
+Assert.assertTrue(ratisPipeline.isPresent());
+Assert.assertTrue(ratisPipeline.get().isHealthy());
+// Verify correct leader info populated
+verifyLeaderInfo(ratisPipeline.get());
+// Verify client connects to Leader without NotLeaderException
+XceiverClientRatis xceiverClientRatis =
+XceiverClientRatis.newXceiverClientRatis(ratisPipeline.get(), conf);
+GenericTestUtils.LogCapturer logCapturer =
+GenericTestUtils.LogCapturer.captureLogs(GrpcClientProtocolClient.LOG);
+xceiverClientRatis.connect();
+logCapturer.stopCapturing();
+Assert.assertFalse("Client should connect to pipeline leader on first
try.",
+logCapturer.getOutput().contains(
+"org.apache.ratis.protocol.NotLeaderException"));
Review comment:
Hi @adoroszlai, good catch, this was from memory, it seems we have since
changed where this gets logged and the level. I have verified by returning
non-leader on the newly added method that we get failure:
`2019-12-10 21:28:23,433 [ForkJoinPool.commonPool-worker-2] DEBUG
client.GrpcClientProtocolService
(GrpcClientProtocolService.java:sendReply(354)) -
0-OrderedRequestStreamObserver0: sendReply seq=1,
RaftClientReply:client-C8FCD8002CD8->1f4d9226-afd7-40f5-9dd3-db27236750e8@group-A49D14340303,
cid=0, FAILED org.apache.ratis.protocol.NotLeaderException: Server
1f4d9226-afd7-40f5-9dd3-db27236750e8@group-A49D14340303 is not the leader
f12ee13d-5e47-4dc1-87ca-f330c773f86f:192.168.1.15:50058, logIndex=0,
commits[1f4d9226-afd7-40f5-9dd3-db27236750e8:c-1,
f12ee13d-5e47-4dc1-87ca-f330c773f86f:c0,
6f12f4a3-71e9-41ff-8fdb-714053a8f799:c0]`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #339: HDDS-2553. Iterator.next() methods should throw NoSuchElementException
xiaoyuyao commented on a change in pull request #339: HDDS-2553.
Iterator.next() methods should throw NoSuchElementException
URL: https://github.com/apache/hadoop-ozone/pull/339#discussion_r356409911
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChunkBufferImplWithByteBuffer.java
##
@@ -53,6 +54,9 @@ public boolean hasNext() {
@Override
public ByteBuffer next() {
final ByteBuffer duplicated = buffer.duplicate();
+if (!buffer.hasRemaining()) {
Review comment:
Should we move this before Line 56? Can you add a unit test for repro and
verification to avoid regression in future?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #339: HDDS-2553. Iterator.next() methods should throw NoSuchElementException
xiaoyuyao commented on a change in pull request #339: HDDS-2553.
Iterator.next() methods should throw NoSuchElementException
URL: https://github.com/apache/hadoop-ozone/pull/339#discussion_r356409911
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/ChunkBufferImplWithByteBuffer.java
##
@@ -53,6 +54,9 @@ public boolean hasNext() {
@Override
public ByteBuffer next() {
final ByteBuffer duplicated = buffer.duplicate();
+if (!buffer.hasRemaining()) {
Review comment:
Change LGTM. Should we move this before Line 56? Can you add a unit test for
repro and verification to avoid regression in future?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cxorm commented on issue #311: HDDS-2532. Sonar : fix issues in OzoneQuota
cxorm commented on issue #311: HDDS-2532. Sonar : fix issues in OzoneQuota URL: https://github.com/apache/hadoop-ozone/pull/311#issuecomment-564385982 Thanks @adoroszlai and @anuengineer for the reviews, and thanks @dineshchitlangia for the commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #332: HDDS-2539. Sonar: Fix sonar warnings in SCMContainerManager class.
xiaoyuyao commented on a change in pull request #332: HDDS-2539. Sonar: Fix sonar warnings in SCMContainerManager class. URL: https://github.com/apache/hadoop-ozone/pull/332#discussion_r356401048 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java ## @@ -79,19 +77,14 @@ * Constructs a mapping class that creates mapping between container names * and pipelines. * - * @param nodeManager - NodeManager so that we can get the nodes that are - * healthy to place new - * containers. * passed to LevelDB and this memory is allocated in Native code space. * CacheSize is specified * in MB. Review comment: missing Javadoc for conf This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2561) Handle InterruptedException in LeaseManager
[ https://issues.apache.org/jira/browse/HDDS-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia resolved HDDS-2561. - Fix Version/s: 0.5.0 Resolution: Fixed > Handle InterruptedException in LeaseManager > --- > > Key: HDDS-2561 > URL: https://issues.apache.org/jira/browse/HDDS-2561 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Priority: Major > Labels: newbie, pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > https://sonarcloud.io/project/issues?id=hadoop-ozone&issues=AW5md-zSKcVY8lQ4ZsJj&open=AW5md-zSKcVY8lQ4ZsJj -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2561) Handle InterruptedException in LeaseManager
[ https://issues.apache.org/jira/browse/HDDS-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993183#comment-16993183 ] Dinesh Chitlangia commented on HDDS-2561: - Thanks [~iamabug] for the contribution. [~aengineer] could you please add Xiang as a contributor? > Handle InterruptedException in LeaseManager > --- > > Key: HDDS-2561 > URL: https://issues.apache.org/jira/browse/HDDS-2561 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Priority: Major > Labels: newbie, pull-request-available, sonar > Time Spent: 20m > Remaining Estimate: 0h > > https://sonarcloud.io/project/issues?id=hadoop-ozone&issues=AW5md-zSKcVY8lQ4ZsJj&open=AW5md-zSKcVY8lQ4ZsJj -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #338: HDDS-2561. Handle InterruptedException in LeaseManager
dineshchitlangia merged pull request #338: HDDS-2561. Handle InterruptedException in LeaseManager URL: https://github.com/apache/hadoop-ozone/pull/338 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2589) Ozone Shell PutKeyHandler inconsistent config override
[
https://issues.apache.org/jira/browse/HDDS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993163#comment-16993163
]
Xiang Zhang commented on HDDS-2589:
---
[~elek] I have replicated this problem, is this intentional ? If not, please
assign this to me [~adoroszlai]
> Ozone Shell PutKeyHandler inconsistent config override
> --
>
> Key: HDDS-2589
> URL: https://issues.apache.org/jira/browse/HDDS-2589
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone CLI
>Reporter: Attila Doroszlai
>Priority: Minor
> Labels: newbie
>
> Ozone Shell allows overriding config file via {{\-conf }} option,
> and also specific settings via {{\-\-set key=value}}. This is partly ignored
> by {{PutKeyHandler}} (which implements {{ozone sh key put}}). By [creating a
> new config
> object|https://github.com/apache/hadoop-ozone/blob/b1ac520ed95fe35b454e55695a05ed71299d8797/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/PutKeyHandler.java#L101]
> instead of using the one from the [factory method
> {{createOzoneConfiguration()}}|https://github.com/apache/hadoop-ozone/blob/b1ac520ed95fe35b454e55695a05ed71299d8797/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/web/ozShell/keys/PutKeyHandler.java#L80],
> it bypasses the config set via command-line. This affects the following
> settings: replication factor and type, chunk size (used only for client-side
> buffer sizing).
> The goal of this task is to first clarify if this is intentional. Then, if it
> is not, the fix is simple:
> * save the object from {{createOzoneConfiguration()}} call to the local
> variable {{conf}}
> * remove {{new OzoneConfiguration()}} call
> To confirm the difference in config override handling:
> * {{ozone sh \-\-set ozone.om.address=no-such-host key put vol1/bucket1/asdf
> /etc/passwd}} fails due to unknown host for OM, so it respects the setting
> * {{ozone sh \-\-set ozone.replication.type=STAND_ALONE key put
> vol1/bucket1/asdf /etc/passwd}} creates a key with RATIS replication despite
> the setting to use STAND_ALONE replication
> CC [~elek] for input on whether this is intentional or can be fixed
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2553) Sonar: Iterator.next() methods should throw NoSuchElementException
[ https://issues.apache.org/jira/browse/HDDS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2553: - Labels: newbie pull-request-available (was: newbie) > Sonar: Iterator.next() methods should throw NoSuchElementException > -- > > Key: HDDS-2553 > URL: https://issues.apache.org/jira/browse/HDDS-2553 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Dinesh Chitlangia >Priority: Major > Labels: newbie, pull-request-available > > [https://sonarcloud.io/project/issues?id=hadoop-ozone&issues=AW6BMujFm2E_7tGaNiTl&open=AW6BMujFm2E_7tGaNiTl] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] iamabug opened a new pull request #339: HDDS-2553. Iterator.next() methods should throw NoSuchElementException
iamabug opened a new pull request #339: HDDS-2553. Iterator.next() methods should throw NoSuchElementException URL: https://github.com/apache/hadoop-ozone/pull/339 ## What changes were proposed in this pull request? throw NoSuchElementException in Iterator.next() ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2553 ## How was this patch tested? Simple change, no test done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
timmylicheng commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356381507
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
##
@@ -128,6 +128,19 @@ public void setPipelineProvider(ReplicationType
replicationType,
pipelineFactory.setProvider(replicationType, provider);
}
+ private int computeNodeIdHash(Pipeline pipeline) {
+if (pipeline.getType() != ReplicationType.RATIS) {
+ return 0;
+}
+
+if (pipeline.getFactor() != ReplicationFactor.THREE) {
+ return 0;
+}
+
+return RatisPipelineUtils.
+encodeNodeIdsOfFactorThreePipeline(pipeline.getNodes());
Review comment:
That's why I sort the datanode list. I will see if XOR could resolve it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2553) Sonar: Iterator.next() methods should throw NoSuchElementException
[ https://issues.apache.org/jira/browse/HDDS-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993152#comment-16993152 ] Xiang Zhang commented on HDDS-2553: --- [~dineshchitlangia] Could you assign this to me ? > Sonar: Iterator.next() methods should throw NoSuchElementException > -- > > Key: HDDS-2553 > URL: https://issues.apache.org/jira/browse/HDDS-2553 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Dinesh Chitlangia >Priority: Major > Labels: newbie > > [https://sonarcloud.io/project/issues?id=hadoop-ozone&issues=AW6BMujFm2E_7tGaNiTl&open=AW6BMujFm2E_7tGaNiTl] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
timmylicheng commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356381388
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -100,4 +105,45 @@ static void destroyPipeline(DatanodeDetails dn,
PipelineID pipelineID,
true, p.getId());
}
}
+
+ static int encodeNodeIdsOfFactorThreePipeline(List nodes) {
+if (nodes.size() != HddsProtos.ReplicationFactor.THREE.getNumber()) {
+ return 0;
+}
+List nodeIds = nodes.stream()
+.map(DatanodeDetails::getUuid).distinct()
+.collect(Collectors.toList());
+nodeIds.sort(new ComparableComparator());
+// Only for Factor THREE pipeline.
+return new HashCodeBuilder()
+.append(nodeIds.get(0).toString())
+.append(nodeIds.get(1).toString())
+.append(nodeIds.get(2).toString())
+.toHashCode();
+ }
+
+ /**
+ * Return first existed pipeline which share the same set of datanodes
+ * with the input pipeline.
+ * @param stateManager PipelineStateManager
+ * @param pipeline input pipeline
+ * @return first matched pipeline
+ */
+ static Pipeline checkPipelineContainSameDatanodes(
Review comment:
I put this into Utils for all classes to access. I tried
PipelineStateManager first and Sammi advised to put it into Utils for some test
classes who don't have pipelineStateManager initialized.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
timmylicheng commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356381056
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
##
@@ -163,6 +177,18 @@ public synchronized Pipeline
createPipeline(ReplicationType type,
metrics.incNumPipelineCreated();
metrics.createPerPipelineMetrics(pipeline);
}
+ Pipeline overlapPipeline = RatisPipelineUtils
+ .checkPipelineContainSameDatanodes(stateManager, pipeline);
+ if (overlapPipeline != null) {
+metrics.incNumPipelineContainSameDatanodes();
+//TODO remove until pipeline allocation is proved equally distributed.
+LOG.info("Pipeline: " + pipeline.getId().toString() +
Review comment:
The log only prints when we find overlapped pipelines who share same
datanodes. From TestPipelineDatanodesIntersection that I observe, overlap shows
up rarely (like 0 in 62 pipelines, 1 in 73 pipelines, 4 in 242 pipelines).
I set up this log to collect overlap cases in regular running in order to
collect information for potential overlapping. My major purpose is to see if
random allocation of datanodes considering rack and topology is good enough to
spread nodes up in pipelines. If the stats proved otherwise, I will try to
implement other allocation algorithm. I talk with Sammi and we feel like we may
not be able to collect enough info in DEBUG mode.
So the log will only print few and while it prints, it could give us a good
idea on how pipelines are spread across all nodes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
timmylicheng commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356379578
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -100,4 +105,45 @@ static void destroyPipeline(DatanodeDetails dn,
PipelineID pipelineID,
true, p.getId());
}
}
+
+ static int encodeNodeIdsOfFactorThreePipeline(List nodes) {
+if (nodes.size() != HddsProtos.ReplicationFactor.THREE.getNumber()) {
+ return 0;
+}
+List nodeIds = nodes.stream()
+.map(DatanodeDetails::getUuid).distinct()
+.collect(Collectors.toList());
+nodeIds.sort(new ComparableComparator());
+// Only for Factor THREE pipeline.
+return new HashCodeBuilder()
+.append(nodeIds.get(0).toString())
+.append(nodeIds.get(1).toString())
+.append(nodeIds.get(2).toString())
+.toHashCode();
+ }
+
+ /**
+ * Return first existed pipeline which share the same set of datanodes
+ * with the input pipeline.
+ * @param stateManager PipelineStateManager
+ * @param pipeline input pipeline
+ * @return first matched pipeline
+ */
+ static Pipeline checkPipelineContainSameDatanodes(
+ PipelineStateManager stateManager, Pipeline pipeline) {
+List matchedPipelines = stateManager.getPipelines(
+HddsProtos.ReplicationType.RATIS,
+HddsProtos.ReplicationFactor.THREE)
+.stream().filter(p -> !p.equals(pipeline) &&
Review comment:
I will use PipelineID then
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2702) Client failed to recover from ratis AlreadyClosedException exception
[
https://issues.apache.org/jira/browse/HDDS-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HDDS-2702:
-
Priority: Blocker (was: Critical)
> Client failed to recover from ratis AlreadyClosedException exception
>
>
> Key: HDDS-2702
> URL: https://issues.apache.org/jira/browse/HDDS-2702
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Priority: Blocker
>
> Run teragen, and it failed to enter the mapreduce stage and print the
> following warnning message on console endlessly.
> {noformat}
> 19/12/10 19:23:54 WARN io.KeyOutputStream: Encountered exception
> java.io.IOException: Unexpected Storage Container Exception:
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException:
> org.apache.ratis.protocol.AlreadyClosedException:
> SlidingWindow$Client:client-FBD45C9313A5->RAFT is closed. on the pipeline
> Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes:
> ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172,
> networkLocation: /rack2, certSerialId:
> null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044,
> networkLocation: /rack2, certSerialId:
> null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122,
> networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE,
> State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last
> committed block length is 0, uncommitted data length is 295833 retry count 0
> 19/12/10 19:23:54 INFO io.BlockOutputStreamEntryPool: Allocating block with
> ExcludeList {datanodes = [], containerIds = [], pipelineIds =
> [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
> 19/12/10 19:26:16 WARN io.KeyOutputStream: Encountered exception
> java.io.IOException: Unexpected Storage Container Exception:
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException:
> org.apache.ratis.protocol.AlreadyClosedException:
> SlidingWindow$Client:client-7C5A7B5CFC31->RAFT is closed. on the pipeline
> Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes:
> ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172,
> networkLocation: /rack2, certSerialId:
> null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044,
> networkLocation: /rack2, certSerialId:
> null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122,
> networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE,
> State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last
> committed block length is 0, uncommitted data length is 295833 retry count 0
> 19/12/10 19:26:16 INFO io.BlockOutputStreamEntryPool: Allocating block with
> ExcludeList {datanodes = [], containerIds = [], pipelineIds =
> [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
> 19/12/10 19:28:38 WARN io.KeyOutputStream: Encountered exception
> java.io.IOException: Unexpected Storage Container Exception:
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException:
> org.apache.ratis.protocol.AlreadyClosedException:
> SlidingWindow$Client:client-B3D8C0052C4E->RAFT is closed. on the pipeline
> Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes:
> ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172,
> networkLocation: /rack2, certSerialId:
> null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044,
> networkLocation: /rack2, certSerialId:
> null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122,
> networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE,
> State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last
> committed block length is 0, uncommitted data length is 295833 retry count 0
> 19/12/10 19:28:38 INFO io.BlockOutputStreamEntryPool: Allocating block with
> ExcludeList {datanodes = [], containerIds = [], pipelineIds =
> [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on issue #117: HDDS-1576. Support configure more than one raft log storage to host m…
ChenSammi commented on issue #117: HDDS-1576. Support configure more than one raft log storage to host m… URL: https://github.com/apache/hadoop-ozone/pull/117#issuecomment-564350141 Hi @mukul1987, would you help to review this patch? The patch is more related with RATIS than Ozone. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2561) Handle InterruptedException in LeaseManager
[ https://issues.apache.org/jira/browse/HDDS-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2561: - Labels: newbie pull-request-available sonar (was: newbie sonar) > Handle InterruptedException in LeaseManager > --- > > Key: HDDS-2561 > URL: https://issues.apache.org/jira/browse/HDDS-2561 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Priority: Major > Labels: newbie, pull-request-available, sonar > > https://sonarcloud.io/project/issues?id=hadoop-ozone&issues=AW5md-zSKcVY8lQ4ZsJj&open=AW5md-zSKcVY8lQ4ZsJj -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] iamabug opened a new pull request #338: HDDS-2561. Handle InterruptedException in LeaseManager
iamabug opened a new pull request #338: HDDS-2561. Handle InterruptedException in LeaseManager URL: https://github.com/apache/hadoop-ozone/pull/338 ## What changes were proposed in this pull request? Re-Interrupt when catching the InterruptedException. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2561 ## How was this patch tested? Simple change, no testing done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2561) Handle InterruptedException in LeaseManager
[ https://issues.apache.org/jira/browse/HDDS-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993104#comment-16993104 ] Xiang Zhang commented on HDDS-2561: --- Newbie here, could you assign this to me ? [~dineshchitlangia] > Handle InterruptedException in LeaseManager > --- > > Key: HDDS-2561 > URL: https://issues.apache.org/jira/browse/HDDS-2561 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Dinesh Chitlangia >Priority: Major > Labels: newbie, sonar > > https://sonarcloud.io/project/issues?id=hadoop-ozone&issues=AW5md-zSKcVY8lQ4ZsJj&open=AW5md-zSKcVY8lQ4ZsJj -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] abhishekaypurohit opened a new pull request #337: Hdds 2706 2
abhishekaypurohit opened a new pull request #337: Hdds 2706 2 URL: https://github.com/apache/hadoop-ozone/pull/337 ## What changes were proposed in this pull request? When revoking SCM certs, DNs local cache is not refreshed. This is the first patch for that functionality. Note: This is not integrated yet so it's safe. However, I am going to add Unit Tests with the next patch. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2706 ## How was this patch tested? mvn clean install This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on issue #148: HDDS-2105. Merge OzoneClientFactory#getRpcClient functions
smengcl commented on issue #148: HDDS-2105. Merge OzoneClientFactory#getRpcClient functions URL: https://github.com/apache/hadoop-ozone/pull/148#issuecomment-564286890 Thanks for reviewing and committing this @hanishakoneru ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2706) Add class CRLCodec - used for certificate revocation list.
Abhishek Purohit created HDDS-2706: -- Summary: Add class CRLCodec - used for certificate revocation list. Key: HDDS-2706 URL: https://issues.apache.org/jira/browse/HDDS-2706 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Abhishek Purohit Assignee: Abhishek Purohit At present, any certificate which gets revoked in SCM, is not transparently notified to DNs and local certificate copy on DNs are not being removed. This is the component for generating CRL for the revoked certificates. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2596) Remove unused private method "createPipeline"
[ https://issues.apache.org/jira/browse/HDDS-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Purohit resolved HDDS-2596. Resolution: Fixed > Remove unused private method "createPipeline" > - > > Key: HDDS-2596 > URL: https://issues.apache.org/jira/browse/HDDS-2596 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Abhishek Purohit >Assignee: Abhishek Purohit >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > > [https://sonarcloud.io/project/issues?id=hadoop-ozone&issues=AW5md_AVKcVY8lQ4ZsWe&open=AW5md_AVKcVY8lQ4ZsWe] > and > https://sonarcloud.io/project/issues?id=hadoop-ozone&issues=AW5md_AVKcVY8lQ4ZsWW&open=AW5md_AVKcVY8lQ4ZsWW -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #331: HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro…
bharatviswa504 commented on a change in pull request #331: HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro… URL: https://github.com/apache/hadoop-ozone/pull/331#discussion_r356277517 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java ## @@ -1077,6 +1077,7 @@ private GetFileStatusResponse getOzoneFileStatus( .setVolumeName(keyArgs.getVolumeName()) .setBucketName(keyArgs.getBucketName()) .setKeyName(keyArgs.getKeyName()) +.setRefreshPipeline(keyArgs.getRefreshPipeline()) Review comment: Do we need to set refreshPipeline to true in list status also? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #331: HDDS-2188. Implement LocatedFileStatus & getFileBlockLocations to pro…
bharatviswa504 commented on a change in pull request #331: HDDS-2188. Implement
LocatedFileStatus & getFileBlockLocations to pro…
URL: https://github.com/apache/hadoop-ozone/pull/331#discussion_r356276973
##
File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
##
@@ -677,6 +677,7 @@ message KeyArgs {
// request type.
optional uint64 modificationTime = 13;
optional bool sortDatanodes = 14;
+optional bool refreshPipeline = 15;
Review comment:
Do we need this flag at all? As from the code it looks like we always want
to return with the latest block information or are there any case we want to
send it to false?. So, in OzoneManagerRequestHandler, we can just set
omKeyArgs.set(true) right?
Similar is done for other requests like lookupFile.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on issue #336: HDDS-2630. NullPointerException in S3g.
xiaoyuyao commented on issue #336: HDDS-2630. NullPointerException in S3g. URL: https://github.com/apache/hadoop-ozone/pull/336#issuecomment-564221167 LGTM, can we add the manual verification to the acceptance test suite? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356248292
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
##
@@ -163,6 +177,18 @@ public synchronized Pipeline
createPipeline(ReplicationType type,
metrics.incNumPipelineCreated();
metrics.createPerPipelineMetrics(pipeline);
}
+ Pipeline overlapPipeline = RatisPipelineUtils
+ .checkPipelineContainSameDatanodes(stateManager, pipeline);
+ if (overlapPipeline != null) {
+metrics.incNumPipelineContainSameDatanodes();
+//TODO remove until pipeline allocation is proved equally distributed.
+LOG.info("Pipeline: " + pipeline.getId().toString() +
Review comment:
can we
1) put this with if (LOG.isDebugEnabled())
2) change the log level to DEBUG
3) change to use parameterized log formatting
to avoid performance penalty from logging as this is could affect write?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356248292
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
##
@@ -163,6 +177,18 @@ public synchronized Pipeline
createPipeline(ReplicationType type,
metrics.incNumPipelineCreated();
metrics.createPerPipelineMetrics(pipeline);
}
+ Pipeline overlapPipeline = RatisPipelineUtils
+ .checkPipelineContainSameDatanodes(stateManager, pipeline);
+ if (overlapPipeline != null) {
+metrics.incNumPipelineContainSameDatanodes();
+//TODO remove until pipeline allocation is proved equally distributed.
+LOG.info("Pipeline: " + pipeline.getId().toString() +
Review comment:
can we put this with if (LOG.isDebugEnabled()) and change the log level to
DEBUG as this is could affect write performance?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356245738
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
##
@@ -128,6 +128,19 @@ public void setPipelineProvider(ReplicationType
replicationType,
pipelineFactory.setProvider(replicationType, provider);
}
+ private int computeNodeIdHash(Pipeline pipeline) {
+if (pipeline.getType() != ReplicationType.RATIS) {
+ return 0;
+}
+
+if (pipeline.getFactor() != ReplicationFactor.THREE) {
+ return 0;
+}
+
+return RatisPipelineUtils.
+encodeNodeIdsOfFactorThreePipeline(pipeline.getNodes());
Review comment:
unordered node list can produce different hash for pipelines with the same
datanodes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356244699
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -100,4 +105,45 @@ static void destroyPipeline(DatanodeDetails dn,
PipelineID pipelineID,
true, p.getId());
}
}
+
+ static int encodeNodeIdsOfFactorThreePipeline(List nodes) {
+if (nodes.size() != HddsProtos.ReplicationFactor.THREE.getNumber()) {
+ return 0;
+}
+List nodeIds = nodes.stream()
+.map(DatanodeDetails::getUuid).distinct()
+.collect(Collectors.toList());
+nodeIds.sort(new ComparableComparator());
+// Only for Factor THREE pipeline.
+return new HashCodeBuilder()
+.append(nodeIds.get(0).toString())
+.append(nodeIds.get(1).toString())
+.append(nodeIds.get(2).toString())
+.toHashCode();
+ }
+
+ /**
+ * Return first existed pipeline which share the same set of datanodes
+ * with the input pipeline.
+ * @param stateManager PipelineStateManager
+ * @param pipeline input pipeline
+ * @return first matched pipeline
+ */
+ static Pipeline checkPipelineContainSameDatanodes(
+ PipelineStateManager stateManager, Pipeline pipeline) {
+List matchedPipelines = stateManager.getPipelines(
+HddsProtos.ReplicationType.RATIS,
+HddsProtos.ReplicationFactor.THREE)
+.stream().filter(p -> !p.equals(pipeline) &&
Review comment:
Need to check the override of equals() in Pipeline object, the list of nodes
may not in the same order. This requires us to use getNodesInOrder instead of
getNodes().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2630) NullPointerException in S3g
[ https://issues.apache.org/jira/browse/HDDS-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-2630: - Status: Patch Available (was: In Progress) > NullPointerException in S3g > --- > > Key: HDDS-2630 > URL: https://issues.apache.org/jira/browse/HDDS-2630 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: S3 >Reporter: Sammi Chen >Assignee: Bharat Viswanadham >Priority: Critical > Labels: pull-request-available > Attachments: hadoop-ozoneadmin-s3g-host158.log > > Time Spent: 10m > Remaining Estimate: 0h > > See attached log file -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #336: HDDS-2630. NullPointerException in S3g.
bharatviswa504 opened a new pull request #336: HDDS-2630. NullPointerException in S3g. URL: https://github.com/apache/hadoop-ozone/pull/336 ## What changes were proposed in this pull request? Fix null pointer exception when creating a bucket ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2630 ## How was this patch tested? Tested in docker-compose ozones3 cluster with out credentials. bash-4.2$ aws s3api --endpoint http://s3g:9878 create-bucket --bucket b12345 Unable to locate credentials. You can configure credentials by running "aws configure". This is the same behavior when running against AWS S3. Used the same error code for S3Gateway also. ** AuthorizationHeaderMalformed The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential. 5E763C5A06D2B716 MY4ON8DQYtZnCj2kQn7OZVQ4bvzeHalbwmZt4Ysq87y63DbBxpaTDTLTuYNFFZ8Ol314nMUTcOI= ** This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2630) NullPointerException in S3g
[ https://issues.apache.org/jira/browse/HDDS-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2630: - Labels: pull-request-available (was: ) > NullPointerException in S3g > --- > > Key: HDDS-2630 > URL: https://issues.apache.org/jira/browse/HDDS-2630 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: S3 >Reporter: Sammi Chen >Assignee: Bharat Viswanadham >Priority: Critical > Labels: pull-request-available > Attachments: hadoop-ozoneadmin-s3g-host158.log > > > See attached log file -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356241293
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -100,4 +105,45 @@ static void destroyPipeline(DatanodeDetails dn,
PipelineID pipelineID,
true, p.getId());
}
}
+
+ static int encodeNodeIdsOfFactorThreePipeline(List nodes) {
+if (nodes.size() != HddsProtos.ReplicationFactor.THREE.getNumber()) {
+ return 0;
+}
+List nodeIds = nodes.stream()
+.map(DatanodeDetails::getUuid).distinct()
+.collect(Collectors.toList());
+nodeIds.sort(new ComparableComparator());
+// Only for Factor THREE pipeline.
+return new HashCodeBuilder()
+.append(nodeIds.get(0).toString())
+.append(nodeIds.get(1).toString())
+.append(nodeIds.get(2).toString())
+.toHashCode();
+ }
+
+ /**
+ * Return first existed pipeline which share the same set of datanodes
+ * with the input pipeline.
+ * @param stateManager PipelineStateManager
+ * @param pipeline input pipeline
+ * @return first matched pipeline
+ */
+ static Pipeline checkPipelineContainSameDatanodes(
Review comment:
Should we move this to a non-static PipelineStateManager method?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356232676
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
##
@@ -100,4 +105,45 @@ static void destroyPipeline(DatanodeDetails dn,
PipelineID pipelineID,
true, p.getId());
}
}
+
+ static int encodeNodeIdsOfFactorThreePipeline(List nodes) {
+if (nodes.size() != HddsProtos.ReplicationFactor.THREE.getNumber()) {
+ return 0;
+}
+List nodeIds = nodes.stream()
+.map(DatanodeDetails::getUuid).distinct()
+.collect(Collectors.toList());
+nodeIds.sort(new ComparableComparator());
Review comment:
Can we use orderless hash here to avoid sorting? E.g., xor the hash of all
the nodeId hash?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out pipeline allocation on datanodes and add metrcs/test
xiaoyuyao commented on a change in pull request #291: HDDS-1574 Average out
pipeline allocation on datanodes and add metrcs/test
URL: https://github.com/apache/hadoop-ozone/pull/291#discussion_r356221673
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java
##
@@ -170,12 +169,18 @@ public Pipeline create(ReplicationFactor factor) throws
IOException {
throw new IllegalStateException("Unknown factor: " + factor.name());
}
+int nodeIdHash = 0;
+if (factor == ReplicationFactor.THREE) {
Review comment:
NIT: this condition check can be combined into the switch(factor) after line
165.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2630) NullPointerException in S3g
[
https://issues.apache.org/jira/browse/HDDS-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992852#comment-16992852
]
Bharat Viswanadham commented on HDDS-2630:
--
{quote} I think if there is auth header missing, we'd better add some check and
return an error code instead of throw out an NPE, which is kind of scary.
{quote}
Agreed, will check the AWS S3 behavior in this case, and correspondingly fix in
a similar manner S3 Gateway.
> NullPointerException in S3g
> ---
>
> Key: HDDS-2630
> URL: https://issues.apache.org/jira/browse/HDDS-2630
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
>Reporter: Sammi Chen
>Assignee: Bharat Viswanadham
>Priority: Critical
> Attachments: hadoop-ozoneadmin-s3g-host158.log
>
>
> See attached log file
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #332: HDDS-2539. Sonar: Fix sonar warnings in SCMContainerManager class.
adoroszlai commented on a change in pull request #332: HDDS-2539. Sonar: Fix
sonar warnings in SCMContainerManager class.
URL: https://github.com/apache/hadoop-ozone/pull/332#discussion_r356211822
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/SCMContainerManager.java
##
@@ -90,8 +85,7 @@
*/
@SuppressWarnings("unchecked")
Review comment:
Only if you need to update the patch anyway: this `SuppressWarnings` is
unnecessary, can be removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #322: HDDS-2684. Refactor common test utilities to hadoop-hdds/common
adoroszlai commented on issue #322: HDDS-2684. Refactor common test utilities to hadoop-hdds/common URL: https://github.com/apache/hadoop-ozone/pull/322#issuecomment-564172503 @xiaoyuyao may I ask you to take another look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #313: HDDS-2242. Avoid unnecessary rpc needed to discover the pipeline leader.
adoroszlai commented on a change in pull request #313: HDDS-2242. Avoid
unnecessary rpc needed to discover the pipeline leader.
URL: https://github.com/apache/hadoop-ozone/pull/313#discussion_r356208296
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestRatisPipelineLeader.java
##
@@ -0,0 +1,124 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdds.scm;
+
+import java.util.List;
+import java.util.Optional;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import org.apache.hadoop.ozone.HddsDatanodeService;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import
org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.ratis.grpc.client.GrpcClientProtocolClient;
+import org.apache.ratis.protocol.ClientId;
+import org.apache.ratis.protocol.GroupInfoReply;
+import org.apache.ratis.protocol.GroupInfoRequest;
+import org.apache.ratis.protocol.RaftGroupId;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Test pipeline leader information is correctly used.
+ */
+public class TestRatisPipelineLeader {
+ private static MiniOzoneCluster cluster;
+ private static OzoneConfiguration conf;
+
+ @BeforeClass
+ public static void setup() throws Exception {
+conf = new OzoneConfiguration();
+cluster = MiniOzoneCluster
+.newBuilder(conf)
+.setNumDatanodes(3)
+.build();
+cluster.waitForClusterToBeReady();
+ }
+
+ @AfterClass
+ public static void shutdown() throws Exception {
+if (cluster != null) {
+ cluster.shutdown();
+}
+ }
+
+ @Test
+ public void testLeaderIdUsedOnFirstCall() throws Exception {
+List pipelines = cluster.getStorageContainerManager()
+.getPipelineManager().getPipelines();
+Optional ratisPipeline = pipelines.stream().filter(p ->
+p.getType().equals(HddsProtos.ReplicationType.RATIS)).findFirst();
+Assert.assertTrue(ratisPipeline.isPresent());
+Assert.assertTrue(ratisPipeline.get().isHealthy());
+// Verify correct leader info populated
+verifyLeaderInfo(ratisPipeline.get());
+// Verify client connects to Leader without NotLeaderException
+XceiverClientRatis xceiverClientRatis =
+XceiverClientRatis.newXceiverClientRatis(ratisPipeline.get(), conf);
+GenericTestUtils.LogCapturer logCapturer =
+GenericTestUtils.LogCapturer.captureLogs(GrpcClientProtocolClient.LOG);
+xceiverClientRatis.connect();
+logCapturer.stopCapturing();
+Assert.assertFalse("Client should connect to pipeline leader on first
try.",
+logCapturer.getOutput().contains(
+"org.apache.ratis.protocol.NotLeaderException"));
Review comment:
Can this assertion ever fail? No matter how I tried setting log level for
`GrpcClientProtocolClient.LOG`, output was always empty. Even changing
`getLeaderNode()` to deliberately return a non-leader node didn't trigger any
log. Am I missing something?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2705) Use Ozone specific LICENCE and NOTICE in the root of the git repo
[ https://issues.apache.org/jira/browse/HDDS-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2705: - Labels: pull-request-available (was: ) > Use Ozone specific LICENCE and NOTICE in the root of the git repo > - > > Key: HDDS-2705 > URL: https://issues.apache.org/jira/browse/HDDS-2705 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Blocker > Labels: pull-request-available > > We have three places where we need the LICENSE/NOTICE files: > > # In the binary tar file (binary specific license) > # In the src tar file (source specific license) > # In the root of the git repository > 1 and 2 are fine now (we have dedicated file under hadoop-ozone/dist) but as > we have shared the repository with main hadoop the root LICENSE/NOTICE (3) > still contains the hadoop specific content. > We need to use the files from (2) for (3). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek opened a new pull request #335: HDDS-2705. Use Ozone specific LICENCE and NOTICE in the root of the git repo
elek opened a new pull request #335: HDDS-2705. Use Ozone specific LICENCE and NOTICE in the root of the git repo URL: https://github.com/apache/hadoop-ozone/pull/335 ## What changes were proposed in this pull request? We have three places where we need the LICENSE/NOTICE files: 1. In the binary tar file (binary specific license) 2. In the src tar file (source specific license) 3. In the root of the git repository 1 and 2 are fine now (we have dedicated file under hadoop-ozone/dist) but as we have shared the repository with main hadoop the root LICENSE/NOTICE (3) still contains the hadoop specific content. We need to use the files from (2) for (3). ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2705 ## How was this patch tested? Check the dist build still works: ``` mvn clean install -Dmaven.javadoc.skip=true -DskipTests -Psign,dist,src -Dtar -Dgpg.keyname=$CODESIGNINGKEY ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2705) Use Ozone specific LICENCE and NOTICE in the root of the git repo
[ https://issues.apache.org/jira/browse/HDDS-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek reassigned HDDS-2705: - Assignee: Marton Elek > Use Ozone specific LICENCE and NOTICE in the root of the git repo > - > > Key: HDDS-2705 > URL: https://issues.apache.org/jira/browse/HDDS-2705 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Blocker > > We have three places where we need the LICENSE/NOTICE files: > > # In the binary tar file (binary specific license) > # In the src tar file (source specific license) > # In the root of the git repository > 1 and 2 are fine now (we have dedicated file under hadoop-ozone/dist) but as > we have shared the repository with main hadoop the root LICENSE/NOTICE (3) > still contains the hadoop specific content. > We need to use the files from (2) for (3). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2682) OM File create request does not check for existing directory with the same name
[ https://issues.apache.org/jira/browse/HDDS-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992756#comment-16992756 ] Anu Engineer commented on HDDS-2682: bq. Sometime back, before moving to new implementations of OMFileCreateRequest and OMDirectoryCreateRequest, the getFileStatus(in KeyManagerImpl) would do a prefix lookup to identify if the directory already exists. This is something that is missing in OMFileCreateRequest#checkKeysUnderPath and this is where the bug is. Now that I have written about this issue very clearly, if you want to get back to the point before HA patches. Please feel free. I am not against it. It is just that I thought we were trying to solve a fundamental problem. And hopefully someone will see this discussion in future to understand the trade-offs we are making. Plus, is this important that we need to rollback to previous behavior ? does that kill as use-cases? > OM File create request does not check for existing directory with the same > name > --- > > Key: HDDS-2682 > URL: https://issues.apache.org/jira/browse/HDDS-2682 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > Assume the following sequence of operations/requests: > Req 1. create file -> /d1/d2/d3/d4/k1 (d3 implicitly is a sub-directory > inside /d1/d2) > Req 2. create file -> /d1/d2/d3 (d3 as a file inside /d1/d2) > When processing request 2, OMFileCreateRequest needs to check if 'd1/d2/d3' > is the name of an existing file or an existing directory. In which case the > request has to fail. > Currently for request 2, OM will check explicitly if there is a key > '/d1/d2/d3' in the key table. > Also for non-recursive create requests, OM will check if parent directory > /d1/d2 already exists. For this, the OM iterates the key table to check if > 'd1/d2' occurs as a prefix of any key in the key table - checkKeysUnderPath() > What is missing in current behaviour? > For OM File create, the table iterator must also determine if '/d1/d2/d3' > exists as a prefix for any key in the key table - not just '/d1/d2'. > This fix is required for the correctness of OzoneFS namespace. There is a > potential performance impact - which is outside the scope of this jira and > will be addressed separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-2682) OM File create request does not check for existing directory with the same name
[ https://issues.apache.org/jira/browse/HDDS-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992756#comment-16992756 ] Anu Engineer edited comment on HDDS-2682 at 12/10/19 5:31 PM: -- bq. Sometime back, before moving to new implementations of OMFileCreateRequest and OMDirectoryCreateRequest, the getFileStatus(in KeyManagerImpl) would do a prefix lookup to identify if the directory already exists. This is something that is missing in OMFileCreateRequest#checkKeysUnderPath and this is where the bug is. Now that I have written about this issue very clearly, if you want to get back to the point before HA patches. Please feel free. I am not against it. It is just that I thought we were trying to solve a fundamental problem. And hopefully someone will see this discussion in future to understand the trade-offs we are making. Plus, is this important that we need to rollback to previous behavior ? does that kill any use-cases? was (Author: anu): bq. Sometime back, before moving to new implementations of OMFileCreateRequest and OMDirectoryCreateRequest, the getFileStatus(in KeyManagerImpl) would do a prefix lookup to identify if the directory already exists. This is something that is missing in OMFileCreateRequest#checkKeysUnderPath and this is where the bug is. Now that I have written about this issue very clearly, if you want to get back to the point before HA patches. Please feel free. I am not against it. It is just that I thought we were trying to solve a fundamental problem. And hopefully someone will see this discussion in future to understand the trade-offs we are making. Plus, is this important that we need to rollback to previous behavior ? does that kill as use-cases? > OM File create request does not check for existing directory with the same > name > --- > > Key: HDDS-2682 > URL: https://issues.apache.org/jira/browse/HDDS-2682 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > Assume the following sequence of operations/requests: > Req 1. create file -> /d1/d2/d3/d4/k1 (d3 implicitly is a sub-directory > inside /d1/d2) > Req 2. create file -> /d1/d2/d3 (d3 as a file inside /d1/d2) > When processing request 2, OMFileCreateRequest needs to check if 'd1/d2/d3' > is the name of an existing file or an existing directory. In which case the > request has to fail. > Currently for request 2, OM will check explicitly if there is a key > '/d1/d2/d3' in the key table. > Also for non-recursive create requests, OM will check if parent directory > /d1/d2 already exists. For this, the OM iterates the key table to check if > 'd1/d2' occurs as a prefix of any key in the key table - checkKeysUnderPath() > What is missing in current behaviour? > For OM File create, the table iterator must also determine if '/d1/d2/d3' > exists as a prefix for any key in the key table - not just '/d1/d2'. > This fix is required for the correctness of OzoneFS namespace. There is a > potential performance impact - which is outside the scope of this jira and > will be addressed separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2704) Source tar file is not created during the relase build
[
https://issues.apache.org/jira/browse/HDDS-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2704:
-
Labels: pull-request-available (was: )
> Source tar file is not created during the relase build
> --
>
> Key: HDDS-2704
> URL: https://issues.apache.org/jira/browse/HDDS-2704
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Blocker
> Labels: pull-request-available
>
> Thanks to [~dineshchitlangia] who reported this problem.
> With a release build:
> {code:java}
> mvn clean install -Dmaven.javadoc.skip=true -DskipTests -Psign,dist,src -Dtar
> -Dgpg.keyname=$CODESIGNINGKEY {code}
> The source package (*the* release) is not created.
> In fact it's created, but the problem with the order of clean and install:
> * clean is executed in the root project
> * install is executed in the root project (creates
> hadoop-ozone/dist/target/..src.tar.gx
> * .
> * clean is executed in the hadoop-ozone/dist project *(here the src package
> is deleted)*
> * install is executed in the hadoop-ozone/dist project
> One possible fix is to move the creation of the src package to the
> hadoop-ozone/dist project (but do it from the project root)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek opened a new pull request #334: HDDS-2704. Source tar file is not created during the relase build
elek opened a new pull request #334: HDDS-2704. Source tar file is not created during the relase build URL: https://github.com/apache/hadoop-ozone/pull/334 ## What changes were proposed in this pull request? Thanks to @dineshchitlangia who reported this problem. With a release build: ``` mvn clean install -Dmaven.javadoc.skip=true -DskipTests -Psign,dist,src -Dtar -Dgpg.keyname=$CODESIGNINGKEY ``` The source package (__the__ release) is not created. In fact it's created, but the problem with the order of clean and install: * clean is executed in the root project * install is executed in the root project (creates hadoop-ozone/dist/target/..src.tar.gx * . * clean is executed in the hadoop-ozone/dist project (here the src package is deleted) * install is executed in the hadoop-ozone/dist project One possible fix is to move the creation of the src package to the hadoop-ozone/dist project (but do it from the project root) ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-2704 ## How was this patch tested? ``` mvn clean install -Dmaven.javadoc.skip=true -DskipTests -Psign,dist,src -Dtar -Dgpg.keyname=$CODESIGNINGKEY ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2682) OM File create request does not check for existing directory with the same name
[ https://issues.apache.org/jira/browse/HDDS-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992754#comment-16992754 ] Anu Engineer commented on HDDS-2682: bq. Reiterating once again, with the new implementations of OMFileCreateRequest and OMDirectoryCreateRequest this is possible just with Filesystem Apis. This is a regression from our earlier code and this needs to be fixed. My own view is that this regression always exists, since you can do this via putkey and always was possible, so it is not a new regression. I was thinking we can take two approaches -- either trust the people will not be doing things like this. That is what we do, when we don't check or control this via putKey. Or we can go all OM and enforce this all over. I believe you are proposing a new approach, that is if we are in file sytem client code, we do this ? and not on the server side ? or are you proposing we do this in the OM server side but only in the File system path ? Perhaps a proper enumeration of all the cases and the definition of the proposed solution might be useful before we write code for this feature. > OM File create request does not check for existing directory with the same > name > --- > > Key: HDDS-2682 > URL: https://issues.apache.org/jira/browse/HDDS-2682 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > Assume the following sequence of operations/requests: > Req 1. create file -> /d1/d2/d3/d4/k1 (d3 implicitly is a sub-directory > inside /d1/d2) > Req 2. create file -> /d1/d2/d3 (d3 as a file inside /d1/d2) > When processing request 2, OMFileCreateRequest needs to check if 'd1/d2/d3' > is the name of an existing file or an existing directory. In which case the > request has to fail. > Currently for request 2, OM will check explicitly if there is a key > '/d1/d2/d3' in the key table. > Also for non-recursive create requests, OM will check if parent directory > /d1/d2 already exists. For this, the OM iterates the key table to check if > 'd1/d2' occurs as a prefix of any key in the key table - checkKeysUnderPath() > What is missing in current behaviour? > For OM File create, the table iterator must also determine if '/d1/d2/d3' > exists as a prefix for any key in the key table - not just '/d1/d2'. > This fix is required for the correctness of OzoneFS namespace. There is a > potential performance impact - which is outside the scope of this jira and > will be addressed separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2705) Use Ozone specific LICENCE and NOTICE in the root of the git repo
Marton Elek created HDDS-2705: - Summary: Use Ozone specific LICENCE and NOTICE in the root of the git repo Key: HDDS-2705 URL: https://issues.apache.org/jira/browse/HDDS-2705 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Marton Elek We have three places where we need the LICENSE/NOTICE files: # In the binary tar file (binary specific license) # In the src tar file (source specific license) # In the root of the git repository 1 and 2 are fine now (we have dedicated file under hadoop-ozone/dist) but as we have shared the repository with main hadoop the root LICENSE/NOTICE (3) still contains the hadoop specific content. We need to use the files from (2) for (3). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-2682) OM File create request does not check for existing directory with the same name
[ https://issues.apache.org/jira/browse/HDDS-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992754#comment-16992754 ] Anu Engineer edited comment on HDDS-2682 at 12/10/19 5:27 PM: -- bq. Reiterating once again, with the new implementations of OMFileCreateRequest and OMDirectoryCreateRequest this is possible just with Filesystem Apis. This is a regression from our earlier code and this needs to be fixed. My own view is that this regression has always existed, since you can do this via putkey and always was possible, so it is not a new regression. I was thinking we can take two approaches -- either trust the people will not be doing things like this. That is what we do, when we don't check or control this via putKey. Or we can go all OM and enforce this all over. I believe you are proposing a new approach, that is if we are in file sytem client code, we do this ? and not on the server side ? or are you proposing we do this in the OM server side but only in the File system path ? Perhaps a proper enumeration of all the cases and the definition of the proposed solution might be useful before we write code for this feature. was (Author: anu): bq. Reiterating once again, with the new implementations of OMFileCreateRequest and OMDirectoryCreateRequest this is possible just with Filesystem Apis. This is a regression from our earlier code and this needs to be fixed. My own view is that this regression always exists, since you can do this via putkey and always was possible, so it is not a new regression. I was thinking we can take two approaches -- either trust the people will not be doing things like this. That is what we do, when we don't check or control this via putKey. Or we can go all OM and enforce this all over. I believe you are proposing a new approach, that is if we are in file sytem client code, we do this ? and not on the server side ? or are you proposing we do this in the OM server side but only in the File system path ? Perhaps a proper enumeration of all the cases and the definition of the proposed solution might be useful before we write code for this feature. > OM File create request does not check for existing directory with the same > name > --- > > Key: HDDS-2682 > URL: https://issues.apache.org/jira/browse/HDDS-2682 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > Assume the following sequence of operations/requests: > Req 1. create file -> /d1/d2/d3/d4/k1 (d3 implicitly is a sub-directory > inside /d1/d2) > Req 2. create file -> /d1/d2/d3 (d3 as a file inside /d1/d2) > When processing request 2, OMFileCreateRequest needs to check if 'd1/d2/d3' > is the name of an existing file or an existing directory. In which case the > request has to fail. > Currently for request 2, OM will check explicitly if there is a key > '/d1/d2/d3' in the key table. > Also for non-recursive create requests, OM will check if parent directory > /d1/d2 already exists. For this, the OM iterates the key table to check if > 'd1/d2' occurs as a prefix of any key in the key table - checkKeysUnderPath() > What is missing in current behaviour? > For OM File create, the table iterator must also determine if '/d1/d2/d3' > exists as a prefix for any key in the key table - not just '/d1/d2'. > This fix is required for the correctness of OzoneFS namespace. There is a > potential performance impact - which is outside the scope of this jira and > will be addressed separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2682) OM File create request does not check for existing directory with the same name
[ https://issues.apache.org/jira/browse/HDDS-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992750#comment-16992750 ] Mukul Kumar Singh commented on HDDS-2682: - The problem with putKey which you explained can certainly happen. I completely agree with that and this jira is not trying to address that. However, it is also important to notice that this something which we can reproduce with just filesystem API without any putKey right now. Sometime back, before moving to new implementations of OMFileCreateRequest and OMDirectoryCreateRequest, the getFileStatus(in KeyManagerImpl) would do a prefix lookup to identify if the directory already exists. This is something that is missing in OMFileCreateRequest#checkKeysUnderPath and this is where the bug is. Reiterating once again, with the new implementations of OMFileCreateRequest and OMDirectoryCreateRequest this is possible just with Filesystem Apis. This is a regression from our earlier code and this needs to be fixed. > OM File create request does not check for existing directory with the same > name > --- > > Key: HDDS-2682 > URL: https://issues.apache.org/jira/browse/HDDS-2682 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > Assume the following sequence of operations/requests: > Req 1. create file -> /d1/d2/d3/d4/k1 (d3 implicitly is a sub-directory > inside /d1/d2) > Req 2. create file -> /d1/d2/d3 (d3 as a file inside /d1/d2) > When processing request 2, OMFileCreateRequest needs to check if 'd1/d2/d3' > is the name of an existing file or an existing directory. In which case the > request has to fail. > Currently for request 2, OM will check explicitly if there is a key > '/d1/d2/d3' in the key table. > Also for non-recursive create requests, OM will check if parent directory > /d1/d2 already exists. For this, the OM iterates the key table to check if > 'd1/d2' occurs as a prefix of any key in the key table - checkKeysUnderPath() > What is missing in current behaviour? > For OM File create, the table iterator must also determine if '/d1/d2/d3' > exists as a prefix for any key in the key table - not just '/d1/d2'. > This fix is required for the correctness of OzoneFS namespace. There is a > potential performance impact - which is outside the scope of this jira and > will be addressed separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2704) Source tar file is not created during the relase build
[
https://issues.apache.org/jira/browse/HDDS-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek reassigned HDDS-2704:
-
Assignee: Marton Elek
> Source tar file is not created during the relase build
> --
>
> Key: HDDS-2704
> URL: https://issues.apache.org/jira/browse/HDDS-2704
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Blocker
>
> Thanks to [~dineshchitlangia] who reported this problem.
> With a release build:
> {code:java}
> mvn clean install -Dmaven.javadoc.skip=true -DskipTests -Psign,dist,src -Dtar
> -Dgpg.keyname=$CODESIGNINGKEY {code}
> The source package (*the* release) is not created.
> In fact it's created, but the problem with the order of clean and install:
> * clean is executed in the root project
> * install is executed in the root project (creates
> hadoop-ozone/dist/target/..src.tar.gx
> * .
> * clean is executed in the hadoop-ozone/dist project *(here the src package
> is deleted)*
> * install is executed in the hadoop-ozone/dist project
> One possible fix is to move the creation of the src package to the
> hadoop-ozone/dist project (but do it from the project root)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2704) Source tar file is not created during the relase build
Marton Elek created HDDS-2704:
-
Summary: Source tar file is not created during the relase build
Key: HDDS-2704
URL: https://issues.apache.org/jira/browse/HDDS-2704
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Marton Elek
Thanks to [~dineshchitlangia] who reported this problem.
With a release build:
{code:java}
mvn clean install -Dmaven.javadoc.skip=true -DskipTests -Psign,dist,src -Dtar
-Dgpg.keyname=$CODESIGNINGKEY {code}
The source package (*the* release) is not created.
In fact it's created, but the problem with the order of clean and install:
* clean is executed in the root project
* install is executed in the root project (creates
hadoop-ozone/dist/target/..src.tar.gx
* .
* clean is executed in the hadoop-ozone/dist project *(here the src package is
deleted)*
* install is executed in the hadoop-ozone/dist project
One possible fix is to move the creation of the src package to the
hadoop-ozone/dist project (but do it from the project root)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-2682) OM File create request does not check for existing directory with the same name
[ https://issues.apache.org/jira/browse/HDDS-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992739#comment-16992739 ] Anu Engineer edited comment on HDDS-2682 at 12/10/19 4:56 PM: -- Thank you for the explanation. My apologies I am still confused a little bit bq. What is missing in current behaviour? bq. For OM File create, the table iterator must also determine if '/d1/d2/d3' exists as a prefix for any key in the key table - not just '/d1/d2'. That is what the original description says, and I understand the proposal is to fix this while createFile is happening. My confusion comes from the fact that you can create this non-acceptable prefix as soon the createFile check is done. So really the problem is not solved if we can create this prefix via another putKey. I am sure I am missing something very fundamental here; I would love to learn what I am missing. was (Author: anu): Thank you for the explanation. My apologies I am still confused a little bit bq. What is missing in current behaviour? For OM File create, the table iterator must also determine if '/d1/d2/d3' exists as a prefix for any key in the key table - not just '/d1/d2'. That is what the original description says, and I understand the proposal is to fix this while createFile is happening. My confusion comes from the fact that you can create this non-acceptable prefix as soon the createFile check is done. So really the problem is not solved if we can create this prefix via another putKey. I am sure I am missing something very fundamental here; I would love to learn what I am missing. > OM File create request does not check for existing directory with the same > name > --- > > Key: HDDS-2682 > URL: https://issues.apache.org/jira/browse/HDDS-2682 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > Assume the following sequence of operations/requests: > Req 1. create file -> /d1/d2/d3/d4/k1 (d3 implicitly is a sub-directory > inside /d1/d2) > Req 2. create file -> /d1/d2/d3 (d3 as a file inside /d1/d2) > When processing request 2, OMFileCreateRequest needs to check if 'd1/d2/d3' > is the name of an existing file or an existing directory. In which case the > request has to fail. > Currently for request 2, OM will check explicitly if there is a key > '/d1/d2/d3' in the key table. > Also for non-recursive create requests, OM will check if parent directory > /d1/d2 already exists. For this, the OM iterates the key table to check if > 'd1/d2' occurs as a prefix of any key in the key table - checkKeysUnderPath() > What is missing in current behaviour? > For OM File create, the table iterator must also determine if '/d1/d2/d3' > exists as a prefix for any key in the key table - not just '/d1/d2'. > This fix is required for the correctness of OzoneFS namespace. There is a > potential performance impact - which is outside the scope of this jira and > will be addressed separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2682) OM File create request does not check for existing directory with the same name
[ https://issues.apache.org/jira/browse/HDDS-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992739#comment-16992739 ] Anu Engineer commented on HDDS-2682: Thank you for the explanation. My apologies I am still confused a little bit bq. What is missing in current behaviour? For OM File create, the table iterator must also determine if '/d1/d2/d3' exists as a prefix for any key in the key table - not just '/d1/d2'. That is what the original description says, and I understand the proposal is to fix this while createFile is happening. My confusion comes from the fact that you can create this non-acceptable prefix as soon the createFile check is done. So really the problem is not solved if we can create this prefix via another putKey. I am sure I am missing something very fundamental here; I would love to learn what I am missing. > OM File create request does not check for existing directory with the same > name > --- > > Key: HDDS-2682 > URL: https://issues.apache.org/jira/browse/HDDS-2682 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > Assume the following sequence of operations/requests: > Req 1. create file -> /d1/d2/d3/d4/k1 (d3 implicitly is a sub-directory > inside /d1/d2) > Req 2. create file -> /d1/d2/d3 (d3 as a file inside /d1/d2) > When processing request 2, OMFileCreateRequest needs to check if 'd1/d2/d3' > is the name of an existing file or an existing directory. In which case the > request has to fail. > Currently for request 2, OM will check explicitly if there is a key > '/d1/d2/d3' in the key table. > Also for non-recursive create requests, OM will check if parent directory > /d1/d2 already exists. For this, the OM iterates the key table to check if > 'd1/d2' occurs as a prefix of any key in the key table - checkKeysUnderPath() > What is missing in current behaviour? > For OM File create, the table iterator must also determine if '/d1/d2/d3' > exists as a prefix for any key in the key table - not just '/d1/d2'. > This fix is required for the correctness of OzoneFS namespace. There is a > potential performance impact - which is outside the scope of this jira and > will be addressed separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2703) OzoneFSInputStream to support ByteBufferReadable
[ https://issues.apache.org/jira/browse/HDDS-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien reassigned HDDS-2703: -- Assignee: YiSheng Lien > OzoneFSInputStream to support ByteBufferReadable > > > Key: HDDS-2703 > URL: https://issues.apache.org/jira/browse/HDDS-2703 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: YiSheng Lien >Priority: Major > > This was found by [~cxorm] via HDDS-2443. > ByteBufferReadable could help certain application performance, such as > Impala. (See HDFS-14111) > Additionally, if we support ByteBufferPositionedReadable, it would benefit > HBase. (see HDFS-3246) > Finally, we should add StreamCapabilities to let client probe for these > abilities. (See HDFS-11644) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2703) OzoneFSInputStream to support ByteBufferReadable
Wei-Chiu Chuang created HDDS-2703: - Summary: OzoneFSInputStream to support ByteBufferReadable Key: HDDS-2703 URL: https://issues.apache.org/jira/browse/HDDS-2703 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Wei-Chiu Chuang This was found by [~cxorm] via HDDS-2443. ByteBufferReadable could help certain application performance, such as Impala. (See HDFS-14111) Additionally, if we support ByteBufferPositionedReadable, it would benefit HBase. (see HDFS-3246) Finally, we should add StreamCapabilities to let client probe for these abilities. (See HDFS-11644) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2532) Sonar : fix issues in OzoneQuota
[ https://issues.apache.org/jira/browse/HDDS-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia resolved HDDS-2532. - Fix Version/s: 0.5.0 Resolution: Fixed [~cxorm] thanks for the contribution. [~sdeka] thanks for reporting the issue [~aengineer] & [~adoroszlai] thanks for reviews. > Sonar : fix issues in OzoneQuota > > > Key: HDDS-2532 > URL: https://issues.apache.org/jira/browse/HDDS-2532 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Supratim Deka >Assignee: YiSheng Lien >Priority: Minor > Labels: pull-request-available, sonar > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Sonar issues : > remove runtime exception from declaration. > https://sonarcloud.io/project/issues?id=hadoop-ozone&issues=AW5md-4NKcVY8lQ4ZsO_&open=AW5md-4NKcVY8lQ4ZsO_ > use primitive boolean expression. > https://sonarcloud.io/project/issues?id=hadoop-ozone&issues=AW5md-4NKcVY8lQ4ZsO-&open=AW5md-4NKcVY8lQ4ZsO- -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #311: HDDS-2532. Sonar : fix issues in OzoneQuota
dineshchitlangia merged pull request #311: HDDS-2532. Sonar : fix issues in OzoneQuota URL: https://github.com/apache/hadoop-ozone/pull/311 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2443) Python client/interface for Ozone
[ https://issues.apache.org/jira/browse/HDDS-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992589#comment-16992589 ] YiSheng Lien commented on HDDS-2443: Fix: we can use *export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$OZONE_HOME/share/ozone/lib/hadoop-ozone-filesystem-lib-current-$OZONE_VERSION.jar* > Python client/interface for Ozone > - > > Key: HDDS-2443 > URL: https://issues.apache.org/jira/browse/HDDS-2443 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Client >Reporter: Li Cheng >Priority: Major > Attachments: Ozone with pyarrow.html, Ozone with pyarrow.odt, > OzoneS3.py > > > Original ideas: item#25 in > [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors] > Ozone Client(Python) for Data Science Notebook such as Jupyter. > # Size: Large > # PyArrow: [https://pypi.org/project/pyarrow/] > # Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API > Impala uses libhdfs > > Path to try: > # s3 interface: Ozone s3 gateway(already supported) + AWS python client > (boto3) > # python native RPC > # pyarrow + libhdfs, which use the Java client under the hood. > # python + C interface of go / rust ozone library. I created POC go / rust > clients earlier which can be improved if the libhdfs interface is not good > enough. [By [~elek]] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2682) OM File create request does not check for existing directory with the same name
[ https://issues.apache.org/jira/browse/HDDS-2682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992588#comment-16992588 ] Mukul Kumar Singh commented on HDDS-2682: - Hey Anu, the problem described here is only with the createFile API and is not related to putKey. > OM File create request does not check for existing directory with the same > name > --- > > Key: HDDS-2682 > URL: https://issues.apache.org/jira/browse/HDDS-2682 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > > Assume the following sequence of operations/requests: > Req 1. create file -> /d1/d2/d3/d4/k1 (d3 implicitly is a sub-directory > inside /d1/d2) > Req 2. create file -> /d1/d2/d3 (d3 as a file inside /d1/d2) > When processing request 2, OMFileCreateRequest needs to check if 'd1/d2/d3' > is the name of an existing file or an existing directory. In which case the > request has to fail. > Currently for request 2, OM will check explicitly if there is a key > '/d1/d2/d3' in the key table. > Also for non-recursive create requests, OM will check if parent directory > /d1/d2 already exists. For this, the OM iterates the key table to check if > 'd1/d2' occurs as a prefix of any key in the key table - checkKeysUnderPath() > What is missing in current behaviour? > For OM File create, the table iterator must also determine if '/d1/d2/d3' > exists as a prefix for any key in the key table - not just '/d1/d2'. > This fix is required for the correctness of OzoneFS namespace. There is a > potential performance impact - which is outside the scope of this jira and > will be addressed separately. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2607) DeadNodeHandler should not remove replica for a dead maintenance node
[ https://issues.apache.org/jira/browse/HDDS-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992504#comment-16992504 ] Stephen O'Donnell commented on HDDS-2607: - The staleNodeHandler logic will do the following for a stale node: 1. Close the containers 2. Close the pipelines The dead node handler will do the same as the above, plus: 3. Remove the containerReplica's from the container manager. As 1 and 2 will already be completed for a node decommissioning or entering maintenance, it will do no harm to repeat it for any node admin state. For 3, we only want to avoid it if the node is IN_MAINTENANCE, so that check can easily be performed in the deadNodeHandler. This means we can trigger the HEALTH events only on the node in the nodeStateManager and not be concerned with the operational state. Similarly, the NonHealthyToHealthyNodeHandler triggers pipeline creation, but the creation logic will avoid nodes which are in_service, so it will do no harm to allow a NonHealthyToHealthyNode event to be triggered for any nodes in maintenance states. > DeadNodeHandler should not remove replica for a dead maintenance node > - > > Key: HDDS-2607 > URL: https://issues.apache.org/jira/browse/HDDS-2607 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Affects Versions: 0.5.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > > Normally, when a node goes dead, the DeadNodeHandler removes all the > containers and replica associated with the node from the ContainerManager. > If a node is IN_MAINTENANCE and goes dead, then we do not want to remove its > replica. They should remain present in the system to prevent the container > being marked as under-replicated. > We also need to consider the case where the node is dead, and then > maintenance expires automatically. In that case, the replica associated with > the node must be removed and the affected containers will become > under-replicated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2702) Client failed to recover from ratis AlreadyClosedException exception
[
https://issues.apache.org/jira/browse/HDDS-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HDDS-2702:
-
Description:
Run teragen, and it failed to enter the mapreduce stage and print the following
warnning message on console endlessly.
{noformat}
19/12/10 19:23:54 WARN io.KeyOutputStream: Encountered exception
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.CompletionException:
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.AlreadyClosedException:
SlidingWindow$Client:client-FBD45C9313A5->RAFT is closed. on the pipeline
Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes:
ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172,
networkLocation: /rack2, certSerialId:
null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044,
networkLocation: /rack2, certSerialId:
null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122,
networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE,
State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed
block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:23:54 INFO io.BlockOutputStreamEntryPool: Allocating block with
ExcludeList {datanodes = [], containerIds = [], pipelineIds =
[PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
19/12/10 19:26:16 WARN io.KeyOutputStream: Encountered exception
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.CompletionException:
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.AlreadyClosedException:
SlidingWindow$Client:client-7C5A7B5CFC31->RAFT is closed. on the pipeline
Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes:
ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172,
networkLocation: /rack2, certSerialId:
null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044,
networkLocation: /rack2, certSerialId:
null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122,
networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE,
State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed
block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:26:16 INFO io.BlockOutputStreamEntryPool: Allocating block with
ExcludeList {datanodes = [], containerIds = [], pipelineIds =
[PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
19/12/10 19:28:38 WARN io.KeyOutputStream: Encountered exception
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.CompletionException:
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.AlreadyClosedException:
SlidingWindow$Client:client-B3D8C0052C4E->RAFT is closed. on the pipeline
Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes:
ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172,
networkLocation: /rack2, certSerialId:
null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044,
networkLocation: /rack2, certSerialId:
null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122,
networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE,
State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed
block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:28:38 INFO io.BlockOutputStreamEntryPool: Allocating block with
ExcludeList {datanodes = [], containerIds = [], pipelineIds =
[PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
{noformat}
was:
Run teragen, and it failed to enter the mapreduce stage and print the following
warnning message on console endlessly.
19/12/10 19:23:54 WARN io.KeyOutputStream: Encountered exception
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.CompletionException:
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.AlreadyClosedException:
SlidingWindow$Client:client-FBD45C9313A5->RAFT is closed. on the pipeline
Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes:
ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172,
networkLocation: /rack2, certSerialId:
null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044,
networkLocation: /rack2, certSerialId:
null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122,
networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE,
State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed
block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:23:54 INFO io.BlockOutputStreamEntryPool: Allocating block with
ExcludeList {datanodes = [], containerIds = [], pipelineIds =
[PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
19/12/10 19:26:16 WARN io.KeyOu
[jira] [Created] (HDDS-2702) Client failed to recover from ratis AlreadyClosedException exception
Sammi Chen created HDDS-2702:
Summary: Client failed to recover from ratis
AlreadyClosedException exception
Key: HDDS-2702
URL: https://issues.apache.org/jira/browse/HDDS-2702
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Sammi Chen
Run teragen, and it failed to enter the mapreduce stage and print the following
warnning message on console endlessly.
19/12/10 19:23:54 WARN io.KeyOutputStream: Encountered exception
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.CompletionException:
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.AlreadyClosedException:
SlidingWindow$Client:client-FBD45C9313A5->RAFT is closed. on the pipeline
Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes:
ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172,
networkLocation: /rack2, certSerialId:
null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044,
networkLocation: /rack2, certSerialId:
null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122,
networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE,
State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed
block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:23:54 INFO io.BlockOutputStreamEntryPool: Allocating block with
ExcludeList {datanodes = [], containerIds = [], pipelineIds =
[PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
19/12/10 19:26:16 WARN io.KeyOutputStream: Encountered exception
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.CompletionException:
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.AlreadyClosedException:
SlidingWindow$Client:client-7C5A7B5CFC31->RAFT is closed. on the pipeline
Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes:
ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172,
networkLocation: /rack2, certSerialId:
null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044,
networkLocation: /rack2, certSerialId:
null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122,
networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE,
State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed
block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:26:16 INFO io.BlockOutputStreamEntryPool: Allocating block with
ExcludeList {datanodes = [], containerIds = [], pipelineIds =
[PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
19/12/10 19:28:38 WARN io.KeyOutputStream: Encountered exception
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.CompletionException:
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.AlreadyClosedException:
SlidingWindow$Client:client-B3D8C0052C4E->RAFT is closed. on the pipeline
Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes:
ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172,
networkLocation: /rack2, certSerialId:
null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044,
networkLocation: /rack2, certSerialId:
null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122,
networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE,
State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed
block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:28:38 INFO io.BlockOutputStreamEntryPool: Allocating block with
ExcludeList {datanodes = [], containerIds = [], pipelineIds =
[PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2630) NullPointerException in S3g
[ https://issues.apache.org/jira/browse/HDDS-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992469#comment-16992469 ] Sammi Chen commented on HDDS-2630: -- Hi [~bharat], the issue happened when we tried to create a bucket through s3g gateway. I think if there is auth header missing, we'd better add some check and return an error code instead of throw out an NPE, which is kind of scary. > NullPointerException in S3g > --- > > Key: HDDS-2630 > URL: https://issues.apache.org/jira/browse/HDDS-2630 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: S3 >Reporter: Sammi Chen >Assignee: Bharat Viswanadham >Priority: Critical > Attachments: hadoop-ozoneadmin-s3g-host158.log > > > See attached log file -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] steveloughran commented on issue #321: HDDS-2685. Fix Rename API in BasicOzoneFileSystem
steveloughran commented on issue #321: HDDS-2685. Fix Rename API in BasicOzoneFileSystem URL: https://github.com/apache/hadoop-ozone/pull/321#issuecomment-563975642 a test for rename under self would round this out. it might be best in AbstractContractRenameTest for all to check for This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] steveloughran commented on a change in pull request #321: HDDS-2685. Fix Rename API in BasicOzoneFileSystem
steveloughran commented on a change in pull request #321: HDDS-2685. Fix Rename
API in BasicOzoneFileSystem
URL: https://github.com/apache/hadoop-ozone/pull/321#discussion_r355962809
##
File path:
hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFsRenameDir.java
##
@@ -98,5 +98,13 @@ public void testRenameDir() throws IOException {
// sub-directories of the renamed directory have also been renamed.
assertTrue("Keys under the renamed direcotry not renamed",
fs.exists(new Path(dest, "sub_dir1")));
+
Review comment:
typo worth fixing just above
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData.
lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData. URL: https://github.com/apache/hadoop-ozone/pull/310#issuecomment-563968948 > I am trying to understand what happens if the writeChunkFuture completes exceptionally? Do we not want to remove the entry from the stateMachineDataMap? Or will marking the stateMachine unhealthy take care of it? @hanishakoneru Making stateMachine unhealthy should be sufficient as it would lead to closing of pipeline. However if we do change that behavior in future we would need to handle cache eviction. I have added comments in the code accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 edited a comment on issue #310: HDDS-2542: Race condition between read and write stateMachineData.
lokeshj1703 edited a comment on issue #310: HDDS-2542: Race condition between read and write stateMachineData. URL: https://github.com/apache/hadoop-ozone/pull/310#issuecomment-563967925 > If the cache is full and entry cannot be added to the map because of no available permits, should we also add a timeout so the writeChunk executor thread gets released and does not block for significantly longer? @bshashikant This should not happen in our case as number of pending requests in Ratis is equal to entry limit for our stateMachineDataCache. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData.
lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData. URL: https://github.com/apache/hadoop-ozone/pull/310#issuecomment-563967925 > If the cache is full and entry cannot be added to the map because of no available permits, should we also add a timeout so the writeChunk executor thread gets released and does not block for significantly longer? This should not happen in our case as number of pending requests in Ratis is equal to entry limit for our stateMachineDataCache. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on a change in pull request #310: HDDS-2542: Race condition between read and write stateMachineData.
lokeshj1703 commented on a change in pull request #310: HDDS-2542: Race
condition between read and write stateMachineData.
URL: https://github.com/apache/hadoop-ozone/pull/310#discussion_r355953385
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java
##
@@ -588,10 +593,14 @@ private ByteString readStateMachineData(
/**
* Reads the Entry from the Cache or loads it back by reading from disk.
*/
- private ByteString getCachedStateMachineData(Long logIndex, long term,
- ContainerCommandRequestProto requestProto) throws ExecutionException {
-return stateMachineDataCache.get(logIndex,
-() -> readStateMachineData(requestProto, term, logIndex));
+ private ByteString getStateMachineData(Long logIndex, long term,
+ ContainerCommandRequestProto requestProto)
+ throws IOException {
+ByteString data = stateMachineDataMap.get(logIndex);
+if (data == null) {
+ data = readStateMachineData(requestProto, term, logIndex);
Review comment:
@mukul1987 I have created HDDS-2701 to address this issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2701) Avoid read from temporary chunk file in datanode
Lokesh Jain created HDDS-2701: - Summary: Avoid read from temporary chunk file in datanode Key: HDDS-2701 URL: https://issues.apache.org/jira/browse/HDDS-2701 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Reporter: Lokesh Jain Assignee: Lokesh Jain Currently we try reading chunk data from the temp file if chunk file does not exist. The fix was added in HDDS-2372 due to race condition between readStateMachineData and writeStateMachineData in ContainerStateMachine. After HDDS-2542 is fixed the read from the temp file can be avoided by making sure that chunk data remains in cache until the chunk file is generated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 edited a comment on issue #310: HDDS-2542: Race condition between read and write stateMachineData.
lokeshj1703 edited a comment on issue #310: HDDS-2542: Race condition between read and write stateMachineData. URL: https://github.com/apache/hadoop-ozone/pull/310#issuecomment-563936046 > I tried it with very low request count limit and checked behavior with `freon ockg`. With `ratis.leader.num.pending.requests=2` and `-t 1` the client could make progress, but with `-t 2` both threads were waiting indefinitely at `BlockOutputStream.waitOnFlushFutures`. The same happened with `ratis.leader.num.pending.requests=4` and `-t 4`. @adoroszlai Here the issue is in Ratis. I will create new jiras and post them in the conversation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData.
lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData. URL: https://github.com/apache/hadoop-ozone/pull/310#issuecomment-563936046 > I tried it with very low request count limit and checked behavior with `freon ockg`. With `ratis.leader.num.pending.requests=2` and `-t 1` the client could make progress, but with `-t 2` both threads were waiting indefinitely at `BlockOutputStream.waitOnFlushFutures`. The same happened with `ratis.leader.num.pending.requests=4` and `-t 4`. @adoroszlai Here the issue is in Ratis. I will create new jiras and post them in the conversation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData.
lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData. URL: https://github.com/apache/hadoop-ozone/pull/310#issuecomment-563935333 @bshashikant I have renamed the classes to Cache and ResourceLimitCache. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData.
lokeshj1703 commented on issue #310: HDDS-2542: Race condition between read and write stateMachineData. URL: https://github.com/apache/hadoop-ozone/pull/310#issuecomment-563935124 @adoroszlai @hanishakoneru @mukul1987 @bshashikant Thanks for reviewing the PR! I have added a commit which addresses the review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #323: HDDS-2686. Use protobuf 3 instead of protobuf 2
mukul1987 commented on a change in pull request #323: HDDS-2686. Use protobuf 3 instead of protobuf 2 URL: https://github.com/apache/hadoop-ozone/pull/323#discussion_r355895654 ## File path: pom.xml ## @@ -149,7 +149,7 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xs -2.5.0 +3.10.0 Review comment: This sounds perfect to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #323: HDDS-2686. Use protobuf 3 instead of protobuf 2
mukul1987 commented on a change in pull request #323: HDDS-2686. Use protobuf 3 instead of protobuf 2 URL: https://github.com/apache/hadoop-ozone/pull/323#discussion_r355895403 ## File path: hadoop-hdds/common/src/main/proto/SCMSecurityProtocol.proto ## @@ -21,6 +21,7 @@ * Please see http://wiki.apache.org/hadoop/Compatibility * for what changes are allowed for a *unstable* .proto interface. */ +syntax = "proto2"; Review comment: Thanks makes sense. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
