[jira] [Commented] (HDDS-3964) Ratis config key mismatch
[
https://issues.apache.org/jira/browse/HDDS-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159708#comment-17159708
]
Attila Doroszlai commented on HDDS-3964:
CC [~sammichen] for backport to 0.6.0.
> Ratis config key mismatch
> -
>
> Key: HDDS-3964
> URL: https://issues.apache.org/jira/browse/HDDS-3964
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Neo Yang
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.7.0
>
>
> Some of the Ratis configurations in integration tests are not applied due to
> mismatch in config keys.
> #
> [Ratis|https://github.com/apache/incubator-ratis/blob/master/ratis-client/src/main/java/org/apache/ratis/client/RaftClientConfigKeys.java#L41-L53]:
> {{raft.client.rpc.watch.request.timeout}}
>
> [Ozone|https://github.com/apache/hadoop-ozone/blob/926048403d115ddcb59ff130e5c46e518874b8aa/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java#L119-L122]:
> {{raft.client.watch.request.timeout}}
> #
> [Ratis|https://github.com/apache/incubator-ratis/blob/4db4f804aa90f9900cda08c79b54a45f80f4213b/ratis-server/src/main/java/org/apache/ratis/server/RaftServerConfigKeys.java#L470-L473]:
> {{raft.server.notification.no-leader.timeout}}
>
> [Ozone|https://github.com/apache/hadoop-ozone/blob/926048403d115ddcb59ff130e5c46e518874b8aa/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisServerConfig.java#L42]:
> {{raft.server.Notification.no-leader.timeout}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai merged pull request #1204: HDDS-3964. Ratis config key mismatch
adoroszlai merged pull request #1204: URL: https://github.com/apache/hadoop-ozone/pull/1204 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1204: HDDS-3964. Ratis config key mismatch
adoroszlai commented on pull request #1204: URL: https://github.com/apache/hadoop-ozone/pull/1204#issuecomment-659887007 Thanks @cku328 and @lokeshj1703 for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3964) Ratis config key mismatch
[
https://issues.apache.org/jira/browse/HDDS-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-3964:
---
Fix Version/s: 0.7.0
> Ratis config key mismatch
> -
>
> Key: HDDS-3964
> URL: https://issues.apache.org/jira/browse/HDDS-3964
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Neo Yang
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.7.0
>
>
> Some of the Ratis configurations in integration tests are not applied due to
> mismatch in config keys.
> #
> [Ratis|https://github.com/apache/incubator-ratis/blob/master/ratis-client/src/main/java/org/apache/ratis/client/RaftClientConfigKeys.java#L41-L53]:
> {{raft.client.rpc.watch.request.timeout}}
>
> [Ozone|https://github.com/apache/hadoop-ozone/blob/926048403d115ddcb59ff130e5c46e518874b8aa/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java#L119-L122]:
> {{raft.client.watch.request.timeout}}
> #
> [Ratis|https://github.com/apache/incubator-ratis/blob/4db4f804aa90f9900cda08c79b54a45f80f4213b/ratis-server/src/main/java/org/apache/ratis/server/RaftServerConfigKeys.java#L470-L473]:
> {{raft.server.notification.no-leader.timeout}}
>
> [Ozone|https://github.com/apache/hadoop-ozone/blob/926048403d115ddcb59ff130e5c46e518874b8aa/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisServerConfig.java#L42]:
> {{raft.server.Notification.no-leader.timeout}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] Simon0806 closed pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Simon0806 closed pull request #1041: URL: https://github.com/apache/hadoop-ozone/pull/1041 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3509) Closing container with unhealthy replica on open pipeline
[ https://issues.apache.org/jira/browse/HDDS-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-3509: - Target Version/s: 0.7.0 (was: 0.6.0) > Closing container with unhealthy replica on open pipeline > - > > Key: HDDS-3509 > URL: https://issues.apache.org/jira/browse/HDDS-3509 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.5.0 >Reporter: Nilotpal Nandi >Assignee: Lokesh Jain >Priority: Major > > When a container replica of an OPEN container is marked as UNHEALTHY, SCM > tries to close the container. > If the pipeline is still healthy, we try to close the container via Ratis. > We could run into a scenario where the datanode which marked the container > replica as UNHEALTHY is the pipeline leader. In such case that datanode > (leader) should process the close container command even though the container > replica is in UNHEALTHY state. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3509) Closing container with unhealthy replica on open pipeline
[ https://issues.apache.org/jira/browse/HDDS-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159682#comment-17159682 ] Xiaoyu Yao commented on HDDS-3509: -- Move to 0.7.0 for comprehensive design on how to close containers on unhealthy pipeline. > Closing container with unhealthy replica on open pipeline > - > > Key: HDDS-3509 > URL: https://issues.apache.org/jira/browse/HDDS-3509 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.5.0 >Reporter: Nilotpal Nandi >Assignee: Lokesh Jain >Priority: Major > > When a container replica of an OPEN container is marked as UNHEALTHY, SCM > tries to close the container. > If the pipeline is still healthy, we try to close the container via Ratis. > We could run into a scenario where the datanode which marked the container > replica as UNHEALTHY is the pipeline leader. In such case that datanode > (leader) should process the close container command even though the container > replica is in UNHEALTHY state. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3907) Topology related acceptance test is flaky
[
https://issues.apache.org/jira/browse/HDDS-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDDS-3907:
-
Target Version/s: 0.7.0 (was: 0.6.0)
> Topology related acceptance test is flaky
> -
>
> Key: HDDS-3907
> URL: https://issues.apache.org/jira/browse/HDDS-3907
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Marton Elek
>Assignee: Xiaoyu Yao
>Priority: Blocker
>
> Examples:
> https://github.com/elek/ozone-build-results/tree/master/2020/06/30/1318/acceptance
> https://github.com/elek/ozone-build-results/tree/master/2020/06/30/1321/acceptance
> https://github.com/elek/ozone-build-results/tree/master/2020/06/30/1334/acceptance
> Some strange errors:
> {code}
> scm_1 | 2020-06-30 19:17:50,787 [RatisPipelineUtilsThread] ERROR
> pipeline.SCMPipelineManager: Failed to create pipeline of type RATIS and
> factor ONE. Exception: Cannot create pipeline of factor 1 using 0 nodes. Used
> 6 nodes. Healthy nodes 6
> scm_1 | 2020-06-30 19:17:50,788 [RatisPipelineUtilsThread] ERROR
> pipeline.SCMPipelineManager: Failed to create pipeline of type RATIS and
> factor THREE. Exception: Pipeline creation failed because nodes are engaged
> in other pipelines and every node can only be engaged in max 2 pipelines.
> Required 3. Found 0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3952) Merge small container
[ https://issues.apache.org/jira/browse/HDDS-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3952: - Summary: Merge small container (was: Make container could be reopened) > Merge small container > - > > Key: HDDS-3952 > URL: https://issues.apache.org/jira/browse/HDDS-3952 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: runzhiwang >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1104: HDDS-3612. Allow mounting bucket under other volume
adoroszlai commented on pull request #1104: URL: https://github.com/apache/hadoop-ozone/pull/1104#issuecomment-659851288 Thanks @arp7, @elek for high-level review, @bharatviswa504 for the detailed reviews, @ChenSammi for testing and committing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3977) Add design doc link of OM HA to ozone docs
[ https://issues.apache.org/jira/browse/HDDS-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-3977. -- Resolution: Duplicate > Add design doc link of OM HA to ozone docs > -- > > Key: HDDS-3977 > URL: https://issues.apache.org/jira/browse/HDDS-3977 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > This Jira is to add design document links of OM HA to ozone docs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3977) Add design doc link of OM HA to ozone docs
Bharat Viswanadham created HDDS-3977: Summary: Add design doc link of OM HA to ozone docs Key: HDDS-3977 URL: https://issues.apache.org/jira/browse/HDDS-3977 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham This Jira is to add design document links of OM HA to ozone docs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3967) Remove leftover debug setting
[ https://issues.apache.org/jira/browse/HDDS-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159664#comment-17159664 ] Bharat Viswanadham commented on HDDS-3967: -- [~Sammi] This is committed only to apache main. I don't see this it is committed to ozone-0.6.0. Do you plan to backport this fix to ozone-0.6.0, as I see the fix version is set to 0.6.0? > Remove leftover debug setting > - > > Key: HDDS-3967 > URL: https://issues.apache.org/jira/browse/HDDS-3967 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.6.0 >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Minor > Labels: pull-request-available > Fix For: 0.6.0 > > > HDDS-3821 > [accidentally|https://github.com/apache/hadoop-ozone/pull/1101#issuecomment-658750232] > set some [log level to > DEBUG|https://github.com/apache/hadoop-ozone/blob/926048403d115ddcb59ff130e5c46e518874b8aa/hadoop-ozone/integration-test/src/test/resources/log4j.properties#L23-L24] > for integration tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1211: HDDS-3969. Add validName check for FileSystem requests
bharatviswa504 commented on a change in pull request #1211:
URL: https://github.com/apache/hadoop-ozone/pull/1211#discussion_r456210475
##
File path:
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
##
@@ -721,7 +723,13 @@ public String pathToKey(Path path) {
path = new Path(workingDir, path);
}
// removing leading '/' char
-String key = path.toUri().getPath().substring(1);
+String key = path.toUri().getPath();
+
+if (OzoneFSUtils.isValidName(key)) {
+ key = path.toUri().getPath().substring(1);
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1210: HDDS-3965. SCM failed to start up for duplicated pipeline detected.
avijayanhwx commented on pull request #1210: URL: https://github.com/apache/hadoop-ozone/pull/1210#issuecomment-659832842 cc @fapifta This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1211: HDDS-3969. Add validName check for FileSystem requests
avijayanhwx commented on a change in pull request #1211:
URL: https://github.com/apache/hadoop-ozone/pull/1211#discussion_r456206292
##
File path:
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
##
@@ -721,7 +723,13 @@ public String pathToKey(Path path) {
path = new Path(workingDir, path);
}
// removing leading '/' char
-String key = path.toUri().getPath().substring(1);
+String key = path.toUri().getPath();
+
+if (OzoneFSUtils.isValidName(key)) {
+ key = path.toUri().getPath().substring(1);
Review comment:
Can we just do this here, instead of toUri().getPath() again?
`LOG.trace("path for key:{} is:{}", key, path);
return key.substring(1);
`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3612) Allow mounting bucket under other volume
[ https://issues.apache.org/jira/browse/HDDS-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3612: - Fix Version/s: 0.6.0 > Allow mounting bucket under other volume > > > Key: HDDS-3612 > URL: https://issues.apache.org/jira/browse/HDDS-3612 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Manager >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Blocker > Labels: Triaged, pull-request-available > Fix For: 0.6.0 > > > Step 2 from S3 [volume mapping design > doc|https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/docs/content/design/ozone-volume-management.md#solving-the-mapping-problem-2-4-from-the-problem-listing]: > Implement a bind mount mechanic which makes it possible to mount any > volume/buckets to the specific "s3" volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3612) Allow mounting bucket under other volume
[ https://issues.apache.org/jira/browse/HDDS-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3612: - Resolution: Fixed Status: Resolved (was: Patch Available) > Allow mounting bucket under other volume > > > Key: HDDS-3612 > URL: https://issues.apache.org/jira/browse/HDDS-3612 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Manager >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Blocker > Labels: Triaged, pull-request-available > Fix For: 0.6.0 > > > Step 2 from S3 [volume mapping design > doc|https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/docs/content/design/ozone-volume-management.md#solving-the-mapping-problem-2-4-from-the-problem-listing]: > Implement a bind mount mechanic which makes it possible to mount any > volume/buckets to the specific "s3" volume. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi merged pull request #1104: HDDS-3612. Allow mounting bucket under other volume
ChenSammi merged pull request #1104: URL: https://github.com/apache/hadoop-ozone/pull/1104 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1104: HDDS-3612. Allow mounting bucket under other volume
ChenSammi commented on pull request #1104: URL: https://github.com/apache/hadoop-ozone/pull/1104#issuecomment-659831676 Thanks @adoroszlai for the contribution and @bharatviswa504 for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1190: HDDS-2770. security/SecurityAcls.md
ChenSammi commented on a change in pull request #1190: URL: https://github.com/apache/hadoop-ozone/pull/1190#discussion_r456203522 ## File path: hadoop-hdds/docs/content/security/SecurityAcls.zh.md ## @@ -0,0 +1,66 @@ +--- +title: "Ozone 访问控制列表" +date: "2019-April-03" +weight: 6 +summary: Ozone 原生的授权模块提供了不需要集成 Ranger 的访问控制列表（ACL）支持。 +icon: transfer +--- + + +Ozone 支持一系列原生 ACL，这些 ACL 可以单独用，也可以和 Ranger 协同使用。如果启用了 Apache Ranger，会先检查 Ranger 中的 ACL，再验证 Ozone 内部的 ACL。 + +Ozone 的 ACL 是 Posix ACL 和 S3 ACL 的超集。 + +ACL 的通用格式为 _对象_:_角色_:_权限_. + +_对象_ 可选的值包括： + +1. **卷** - 一个 Ozone 卷，比如 _/volume_ +2. **桶** - 一个 Ozone 桶，比如 _/volume/bucket_ +3. **键** - 一个对象键，比如 _/volume/bucket/key_ +4. **前缀** - 某个键的路径前缀，比如 _/volume/bucket/prefix1/prefix2_ + +_角色_ 可选的值包括: + +1. **用户** - 一个 Kerberos 用户，和 Posix 用户一样，用户可以是已创建的也可以是未创建的。 Review comment: I see. Thanks @xiaoyuyao for the explanation. Then Xiang's translation is more accurate. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1185: HDDS-3933. Fix memory leak because of too many Datanode State Machine Thread
ChenSammi commented on a change in pull request #1185:
URL: https://github.com/apache/hadoop-ozone/pull/1185#discussion_r456200241
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##
@@ -415,7 +431,14 @@ public void execute(ExecutorService service, long time,
TimeUnit unit)
if (this.isEntering()) {
task.onEnter();
}
- task.execute(service);
+
+ if (isThreadPoolAvailable(service)) {
+task.execute(service);
+ } else {
+LOG.warn("No available thread in pool, duration:" +
Review comment:
There is a EndpointStateMachine#logIfNeeded function. We can leverage
this function to reduce the same warning logs logged. @runzhiwang
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1185: HDDS-3933. Fix memory leak because of too many Datanode State Machine Thread
runzhiwang commented on a change in pull request #1185:
URL: https://github.com/apache/hadoop-ozone/pull/1185#discussion_r456195255
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##
@@ -415,7 +431,14 @@ public void execute(ExecutorService service, long time,
TimeUnit unit)
if (this.isEntering()) {
task.onEnter();
}
- task.execute(service);
+
+ if (isThreadPoolAvailable(service)) {
+task.execute(service);
+ } else {
+LOG.warn("No available thread in pool, duration:" +
Review comment:
@xiaoyuyao Thanks for review and reasonable suggestion. How about add
metrics for thread pool available and unavailable times ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3946) s3g multi-upload failed with partName check
[
https://issues.apache.org/jira/browse/HDDS-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HDDS-3946:
-
Target Version/s: 0.7.0 (was: 0.6.0)
> s3g multi-upload failed with partName check
> ---
>
> Key: HDDS-3946
> URL: https://issues.apache.org/jira/browse/HDDS-3946
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Priority: Major
>
> LOGs in S3g,
> INVALID_PART org.apache.hadoop.ozone.om.exceptions.OMException: Complete
> Multipart Upload Failed: volume: s325d55ad283aa400af464c76d713c07adbucket:
> konajdk-profilerkey:
> root@10.121.81.124/dfbd3e3905c34d73ac76ebae5650b7d3.hprof. Provided Part info
> is {
> /s325d55ad283aa400af464c76d713c07ad/konajdk-profiler/root@10.121.81.124/dfbd3e3905c34d73ac76ebae5650b7d3.hprof104478123544835723,
> 200}, where as OM has partName
> /s325d55ad283aa400af464c76d713c07ad/konajdk-profiler/root@10.121.81.124/dfbd3e3905c34d73ac76ebae5650b7d3.hprof104478114690135525
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:589)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.completeMultipartUpload(OzoneManagerProtocolClientSideTranslatorPB.java:884)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.completeMultipartUpload(RpcClient.java:900)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.completeMultipartUpload(OzoneBucket.java:446)
> at
> org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.completeMultipartUpload(ObjectEndpoint.java:476)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> The investigation shows that partNumber 200 block is committed twice with
> different partName, because the ClientID for two commit is different.
> 2020-07-08 20:00:50,087 | INFO | OMAudit | user=root | ip=100.76.18.99 |
> op=COMMIT_MULTIPART_UPLOAD_PARTKEY
> {volume=s325d55ad283aa400af464c76d713c07ad, bucket=konajdk-profiler,
> key=root@10.121.81.124/dfbd3e3905c34d73ac76ebae5650b7d3.hprof,
> dataSize=104857600, replicationType=RATIS, replicationFactor=ONE,
> partNumber=200,
> partName=/s325d55ad283aa400af464c76d713c07ad/konajdk-profiler/root@10.121.81.124/dfbd3e3905c34d73ac76ebae5650b7d3.hprof104478123544835723}
> | ret=SUCCESS |
>
> 2020-07-08 20:01:08,970 | INFO | OMAudit | user=root | ip=100.76.18.99 |
> op=COMMIT_MULTIPART_UPLOAD_PARTKEY
> {volume=s325d55ad283aa400af464c76d713c07ad, bucket=konajdk-profiler,
> key=root@10.121.81.124/dfbd3e3905c34d73ac76ebae5650b7d3.hprof,
> dataSize=104857600, replicationType=RATIS, replicationFactor=ONE,
> partNumber=200,
> partName=/s325d55ad283aa400af464c76d713c07ad/konajdk-profiler/root@10.121.81.124/dfbd3e3905c34d73ac76ebae5650b7d3.hprof104478114690135525}
> | ret=SUCCESS |
> So the question is, can we loose the partName check for this case？
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3952) Make container could be reopened
[ https://issues.apache.org/jira/browse/HDDS-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159636#comment-17159636 ] runzhiwang edited comment on HDDS-3952 at 7/17/20, 3:02 AM: [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small containers which are not full but closed, as the image shows, because ratis pipeline is not stable. So we want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. Then put the containers of map.get(pipeline.datanodes()) on the pipeline and reopen them. [~msingh] What do you think ? !screenshot-1.png! was (Author: yjxxtd): [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small containers which are not full but closed, as the image shows, because ratis pipeline is not stable. So we want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. [~msingh] What do you think ? !screenshot-1.png! > Make container could be reopened > > > Key: HDDS-3952 > URL: https://issues.apache.org/jira/browse/HDDS-3952 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: runzhiwang >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1033: HDDS-3667. If we gracefully stop datanode it would be better to notify scm and r…
maobaolong commented on pull request #1033: URL: https://github.com/apache/hadoop-ozone/pull/1033#issuecomment-659808668 @leosunli After you are ready for review, please use `/ready` to restore this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3952) Make container could be reopened
[ https://issues.apache.org/jira/browse/HDDS-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159636#comment-17159636 ] runzhiwang edited comment on HDDS-3952 at 7/17/20, 3:01 AM: [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small containers which are not full but closed, as the image shows, because ratis pipeline is not stable. So we want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. [~msingh] What do you think ? !screenshot-1.png! was (Author: yjxxtd): [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small containers which are not full but closed, as the image shows, because ratis pipeline is not stable. So We want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. [~msingh] What do you think ? !screenshot-1.png! > Make container could be reopened > > > Key: HDDS-3952 > URL: https://issues.apache.org/jira/browse/HDDS-3952 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: runzhiwang >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3952) Make container could be reopened
[ https://issues.apache.org/jira/browse/HDDS-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159636#comment-17159636 ] runzhiwang edited comment on HDDS-3952 at 7/17/20, 3:00 AM: [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small containers which are not full but closed, as the image shows, because ratis pipeline is not stable. So We want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. [~msingh] What do you think ? !screenshot-1.png! was (Author: yjxxtd): [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small container which is not full but closed, as the image shows, because ratis pipeline is not stable. So We want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. [~msingh] What do you think ? !screenshot-1.png! > Make container could be reopened > > > Key: HDDS-3952 > URL: https://issues.apache.org/jira/browse/HDDS-3952 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: runzhiwang >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3952) Make container could be reopened
[ https://issues.apache.org/jira/browse/HDDS-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159636#comment-17159636 ] runzhiwang commented on HDDS-3952: -- [~msingh] Hi, Thanks for review. The backgroud of this jira is our production cluster has about 7000 small container which is not full but closed, as the image shows, because ratis pipeline is not stable. So We want to reopen and write to the closed but not full container. The basic idea: 1. SCM build a map with entry <3 datanodes, set of closed but not full containers on the 3 datanodes> 2. SCM check whether any open pipeline locate on the 3 datanodes from the map.entrySet, if exists such open pipeline, we get the closed but not full containers by map.get(pipeline.datanodes()), put them on the pipeline and reopen them. 3. When SCM create new pipeline, we first select from the map which 3 datanodes has the most closed but not full containers, and create pipeline on this 3 datanodes. [~msingh] What do you think ? !screenshot-1.png! > Make container could be reopened > > > Key: HDDS-3952 > URL: https://issues.apache.org/jira/browse/HDDS-3952 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: runzhiwang >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3952) Make container could be reopened
[ https://issues.apache.org/jira/browse/HDDS-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3952: - Attachment: screenshot-1.png > Make container could be reopened > > > Key: HDDS-3952 > URL: https://issues.apache.org/jira/browse/HDDS-3952 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode, SCM >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: runzhiwang >Priority: Major > Attachments: screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on pull request #1164: HDDS-3824: OM read requests should make SCM#refreshPipeline outside BUCKET_LOCK
rakeshadr commented on pull request #1164: URL: https://github.com/apache/hadoop-ozone/pull/1164#issuecomment-659786368 Thanks @xiaoyuyao and @bharatviswa504 for the reviews and committing the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on pull request #1149: HDDS-3878. Make OMHA serviceID optional if one (but only one) is defined in the config
xiaoyuyao commented on pull request #1149: URL: https://github.com/apache/hadoop-ozone/pull/1149#issuecomment-659766847 I agree with @elek that the scope of this change is very limited to where client only has one om service id configured. So accidentally access wrong om cluster when multiple om service ids are defined are not a major issue here. There are few case that specify om service id explicitly is helpful even only one om service id is configured. The client specifies different configurations to access two clusters (each with only one service id), e.g., DR(distcp). If we don't require explicit om service id in the uri, the only way to differentiate is the name configuration file locations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1190: HDDS-2770. security/SecurityAcls.md
xiaoyuyao commented on a change in pull request #1190: URL: https://github.com/apache/hadoop-ozone/pull/1190#discussion_r456156802 ## File path: hadoop-hdds/docs/content/security/SecurityAcls.zh.md ## @@ -0,0 +1,66 @@ +--- +title: "Ozone 访问控制列表" +date: "2019-April-03" +weight: 6 +summary: Ozone 原生的授权模块提供了不需要集成 Ranger 的访问控制列表（ACL）支持。 +icon: transfer +--- + + +Ozone 支持一系列原生 ACL，这些 ACL 可以单独用，也可以和 Ranger 协同使用。如果启用了 Apache Ranger，会先检查 Ranger 中的 ACL，再验证 Ozone 内部的 ACL。 + +Ozone 的 ACL 是 Posix ACL 和 S3 ACL 的超集。 + +ACL 的通用格式为 _对象_:_角色_:_权限_. + +_对象_ 可选的值包括： + +1. **卷** - 一个 Ozone 卷，比如 _/volume_ +2. **桶** - 一个 Ozone 桶，比如 _/volume/bucket_ +3. **键** - 一个对象键，比如 _/volume/bucket/key_ +4. **前缀** - 某个键的路径前缀，比如 _/volume/bucket/prefix1/prefix2_ + +_角色_ 可选的值包括: + +1. **用户** - 一个 Kerberos 用户，和 Posix 用户一样，用户可以是已创建的也可以是未创建的。 Review comment: I think it means user/group may or may not have been created in OS/LDAP at the time when you assign them to ozone acl. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1190: HDDS-2770. security/SecurityAcls.md
xiaoyuyao commented on a change in pull request #1190: URL: https://github.com/apache/hadoop-ozone/pull/1190#discussion_r456156398 ## File path: hadoop-hdds/docs/content/security/SecurityAcls.zh.md ## @@ -0,0 +1,66 @@ +--- +title: "Ozone 访问控制列表" +date: "2019-April-03" +weight: 6 +summary: Ozone 原生的授权模块提供了不需要集成 Ranger 的访问控制列表（ACL）支持。 +icon: transfer +--- + + +Ozone 支持一系列原生 ACL，这些 ACL 可以单独用，也可以和 Ranger 协同使用。如果启用了 Apache Ranger，会先检查 Ranger 中的 ACL，再验证 Ozone 内部的 ACL。 Review comment: This is not correct in EN doc. "这些 ACL 可以单独用，也可以和 Ranger 协同使用。如果启用了 Apache Ranger，会先检查 Ranger 中的 ACL，再验证 Ozone 内部的 ACL。" "These ACLs can be used independently or along with Ranger. If Apache Ranger is enabled, then ACL will be checked first with Ranger and then Ozone's internal ACLs will be evaluated." => "These ACLs can be used independently of ozone ACL plugin such as Ranger. If Apache Ranger plugin for Ozone is enabled, then ACL will be checked with Ranger." This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1185: HDDS-3933. Fix memory leak because of too many Datanode State Machine Thread
xiaoyuyao commented on a change in pull request #1185:
URL: https://github.com/apache/hadoop-ozone/pull/1185#discussion_r456153310
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/StateContext.java
##
@@ -415,7 +431,14 @@ public void execute(ExecutorService service, long time,
TimeUnit unit)
if (this.isEntering()) {
task.onEnter();
}
- task.execute(service);
+
+ if (isThreadPoolAvailable(service)) {
+task.execute(service);
+ } else {
+LOG.warn("No available thread in pool, duration:" +
Review comment:
In a similar case of endpoint task locked up, will we end up with
flooding of warning logs here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3969) Validate KeyNames created in FileSystem requests.
[ https://issues.apache.org/jira/browse/HDDS-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3969: - Labels: pull-request-available (was: ) > Validate KeyNames created in FileSystem requests. > - > > Key: HDDS-3969 > URL: https://issues.apache.org/jira/browse/HDDS-3969 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > This jira is to validate KeyNames which are created with OzoneFileSystem. > Similar to how hdfs handles using DFSUtil. isValidName() -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #1211: HDDS-3969. Add validName check for FileSystem requests
bharatviswa504 opened a new pull request #1211: URL: https://github.com/apache/hadoop-ozone/pull/1211 ## What changes were proposed in this pull request? This Jira is to validate KeyNames which are created with OzoneFileSystem. Similar to how hdfs handles using DFSUtil. isValidName() Made only changes in the client. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3969 ## How was this patch tested? Added tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3969) Validate KeyNames created in FileSystem requests.
[ https://issues.apache.org/jira/browse/HDDS-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-3969: - Summary: Validate KeyNames created in FileSystem requests. (was: Validate KeyNames created using KeyCreate and FileCreate) > Validate KeyNames created in FileSystem requests. > - > > Key: HDDS-3969 > URL: https://issues.apache.org/jira/browse/HDDS-3969 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > This jira is to validate KeyNames which are created with OzoneFileSystem. > Similar to how hdfs handles using DFSUtil. isValidName() -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.
bharatviswa504 edited a comment on pull request #1196: URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-659696446 Thank You @arp7 and @avijayanhwx for the review. I have addressed review comments. This PR only modifies behavior of KeyCreate when ozone.om.enable.filesystem.paths is enabled. (Not changed the default behavior of actual KeyCreate) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.
bharatviswa504 edited a comment on pull request #1196: URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-659696446 Thank You @arp7 and @avijayanhwx for the review. I have addressed review comments. This PR only handles KeyCreate when ozone.om.enable.filesystem.paths is enabled. (Not changed behavior of actual KeyCreate) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.
bharatviswa504 edited a comment on pull request #1196: URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-659696446 Thank You @arp7 and @avijayanhwx for the review. I have addressed review comments. This PR only modifies behavior of KeyCreate when ozone.om.enable.filesystem.paths is enabled. (Not changed behavior of actual KeyCreate) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.
bharatviswa504 commented on pull request #1196: URL: https://github.com/apache/hadoop-ozone/pull/1196#issuecomment-659696446 Thank You @arp7 and @avijayanhwx for the review. I have addressed review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3924) Move away from proto serialization for RocksDB keys in Ozone
[ https://issues.apache.org/jira/browse/HDDS-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan resolved HDDS-3924. - Resolution: Fixed > Move away from proto serialization for RocksDB keys in Ozone > > > Key: HDDS-3924 > URL: https://issues.apache.org/jira/browse/HDDS-3924 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Blocker > > Currently there are a few places where we rely on protobuf serialization to > convert the key type to byte array for a RocksDB column family 'key'. > However, from the proto > [documentation|https://developers.google.com/protocol-buffers/docs/encoding#implications] > it looks like the serialization of proto is unreliable (especially across > versions). If the byte[] got from proto serialization changes, this will be a > problem since old keys will be unreadable. > Thanks to [~arp] who helped unearth this problem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3924) Move away from proto serialization for RocksDB keys in Ozone
[ https://issues.apache.org/jira/browse/HDDS-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159496#comment-17159496 ] Aravindan Vijayan commented on HDDS-3924: - Thank you [~pifta], [~ppogde] & [~xyao]. Closing this umbrella JIRA. > Move away from proto serialization for RocksDB keys in Ozone > > > Key: HDDS-3924 > URL: https://issues.apache.org/jira/browse/HDDS-3924 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Blocker > > Currently there are a few places where we rely on protobuf serialization to > convert the key type to byte array for a RocksDB column family 'key'. > However, from the proto > [documentation|https://developers.google.com/protocol-buffers/docs/encoding#implications] > it looks like the serialization of proto is unreliable (especially across > versions). If the byte[] got from proto serialization changes, this will be a > problem since old keys will be unreadable. > Thanks to [~arp] who helped unearth this problem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3926) Ozone Manager Token Identifier table should use in-house serialization rather than rely on proto serialization for key.
[ https://issues.apache.org/jira/browse/HDDS-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-3926. -- Fix Version/s: 0.6.0 Resolution: Fixed Thanks [~ppogde] for the contribution and all for the reviews. Change has been merged and cherry-picked to ozone-0.6.0 > Ozone Manager Token Identifier table should use in-house serialization rather > than rely on proto serialization for key. > --- > > Key: HDDS-3926 > URL: https://issues.apache.org/jira/browse/HDDS-3926 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Aravindan Vijayan >Assignee: Prashant Pogde >Priority: Major > Labels: pull-request-available, upgrade-p0 > Fix For: 0.6.0 > > > Relying on Protobuf serialization for exact match is unreliable according to > the docs. Hence, we have to move away from using proto.toByteArray() for on > disk RocksDB keys. For more details, check parent JIRA. > In the case of the Ozone token identifier, it can be uniquely identified > using the following fields. > * Issue Date > * Master Key ID > * Sequence Number > We can provide a simplified serialization method using the above 3 fields (in > that order) to be used in the Token Identifier codec. > cc [~xyao] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao merged pull request #1182: HDDS-3926. OM Token Identifier table should use in-house serialization.
xiaoyuyao merged pull request #1182: URL: https://github.com/apache/hadoop-ozone/pull/1182 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3824) OM read requests should make SCM#refreshPipeline outside the BUCKET_LOCK
[
https://issues.apache.org/jira/browse/HDDS-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-3824.
--
Fix Version/s: 0.7.0
Resolution: Fixed
Thanks [~rakeshr] for the contribution and all for the reviews. PR has been
merged to master.
> OM read requests should make SCM#refreshPipeline outside the BUCKET_LOCK
> -
>
> Key: HDDS-3824
> URL: https://issues.apache.org/jira/browse/HDDS-3824
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Rakesh Radhakrishnan
>Assignee: Rakesh Radhakrishnan
>Priority: Major
> Labels: om-perf, perfomance, pull-request-available
> Fix For: 0.7.0
>
>
> Refresh pipeline info does a call to SCM and it can be moved outside the
> {{BUCKET_LOCK}}, this would help to improve the performance of read/write mix
> workloads.
> *KeyManagerImpl.java*
> {code:java}
> metadataManager.getLock().acquireReadLock(BUCKET_LOCK, volumeName,
> bucketName);
> try {
> .
> .
> // No need to check if a key is deleted or not here, this is handled
> // when adding entries to cacheKeyMap from DB.
> if (args.getRefreshPipeline()) {
> refreshPipeline(entry.getValue().getKeyInfo());
> }
> .
> .
> } finally {
> metadataManager.getLock().releaseReadLock(BUCKET_LOCK, volumeName,
> bucketName);
> }
> {code}
>
> Code Reference:
> [KeyManagerImpl.java#L2071|https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L2071]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao merged pull request #1164: HDDS-3824: OM read requests should make SCM#refreshPipeline outside BUCKET_LOCK
xiaoyuyao merged pull request #1164: URL: https://github.com/apache/hadoop-ozone/pull/1164 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3976) KeyValueBlockIterator#nextBlock skips valid blocks
[ https://issues.apache.org/jira/browse/HDDS-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Rose updated HDDS-3976: - Description: HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced another one in KeyValueBlockIterator#nextBlock, which depends on the behavior of that method. When the first key encountered does not pass the filter, the internal nextBlock field is never intialized. Then a call to nextBlock() results in call to hasNext() which returns true, which recursively calls nextBlock(), again calling hasNext(), etc until the end of the set is reached and an exception is thrown. This skips all valid keys that may occur past the first invalid key. This bug was identified while working on HDDS-3869, which adds a strong typing layer before objects are serialized into RocksDB for datanode. Due to RocksDB internals, this changes the database layout so that all prefixed keys are returned at the beginning of the key set, instead of in the end. Since the original layout returned all prefixed keys at the end of the key set, this bug was not evident in any of the original unit tests, since the behavior described above could not occur. was: HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced another one in KeyValueBlockIterator#nextBlock, which depends on the behavior of that method. When the first key encountered does not pass the filter, the internal nextBlock field is never intialized. Then a call to nextBlock() results in call to hasNext() which returns true, which recursively calls nextBlock(), again calling hasNext(), etc until the end of the set is reached and an exception is thrown. This skips all valid keys that may occur past the first invalid key. This bug was identified while working on HDDS-3869, which adds a strong typing layer before objects are serialized into RocksDB for datanode. Due to RocksDB internals, this changes the database layout so that all prefixed keys are returned at the beginning of the key set, instead of in the end. Since the original layout returned all prefixed keys at the end of the key set, this bug was not evident in any of the original unit tests, since the behavior described above could not occur. Additionally, the current implementation modifies the internal iterator's state on hasNext() calls, so multiple consecutive hasNext() calls will actually skip over elements. While this is not necessarily a bug, it is atypical iterator behavior and should be resolved to prevent user error. > KeyValueBlockIterator#nextBlock skips valid blocks > -- > > Key: HDDS-3976 > URL: https://issues.apache.org/jira/browse/HDDS-3976 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Major > > HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced > another one in KeyValueBlockIterator#nextBlock, which depends on the behavior > of that method. When the first key encountered does not pass the filter, the > internal nextBlock field is never intialized. Then a call to nextBlock() > results in call to hasNext() which returns true, which recursively calls > nextBlock(), again calling hasNext(), etc until the end of the set is reached > and an exception is thrown. This skips all valid keys that may occur past the > first invalid key. > This bug was identified while working on HDDS-3869, which adds a strong > typing layer before objects are serialized into RocksDB for datanode. Due to > RocksDB internals, this changes the database layout so that all prefixed keys > are returned at the beginning of the key set, instead of in the end. Since > the original layout returned all prefixed keys at the end of the key set, > this bug was not evident in any of the original unit tests, since the > behavior described above could not occur. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3976) KeyValueBlockIterator#nextBlock skips valid blocks
[ https://issues.apache.org/jira/browse/HDDS-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Rose updated HDDS-3976: - Description: HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced another one in KeyValueBlockIterator#nextBlock, which depends on the behavior of that method. When the first key encountered does not pass the filter, the internal nextBlock field is never intialized. Then a call to nextBlock() results in call to hasNext() which returns true, which recursively calls nextBlock(), again calling hasNext(), etc until the end of the set is reached and an exception is thrown. This skips all valid keys that may occur past the first invalid key. This bug was identified while working on HDDS-3869, which adds a strong typing layer before objects are serialized into RocksDB for datanode. Due to RocksDB internals, this changes the database layout so that all prefixed keys are returned at the beginning of the key set, instead of in the end. Since the original layout returned all prefixed keys at the end of the key set, this bug was not evident in any of the original unit tests, since the behavior described above could not occur. Additionally, the current implementation modifies the internal iterator's state on hasNext() calls, so multiple consecutive hasNext() calls will actually skip over elements. While this is not necessarily a bug, it is atypical iterator behavior and should be resolved to prevent user error. was: HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced another one in KeyValueBlockIterator#nextBlock, which depends on the behavior of that method. When the first key encountered does not pass the filter, the internal nextBlock field is never intialized. Then a call to nextBlock() results in call to hasNext() which returns true, which recursively calls nextBlock() etc until the end of the set is reached and an exception is thrown. This skips all valid keys that may occur past the first invalid key. This bug was identified while working on HDDS-3869, which adds a strong typing layer before objects are serialized into RocksDB for datanode. Due to RocksDB internals, this changes the database layout so that all prefixed keys are returned at the beginning of the key set, instead of in the end. Since the original layout returned all prefixed keys at the end of the key set, this bug was not evident in any of the original unit tests, since the behavior described above could not occur. Additionally, the current implementation modifies the internal iterator's state on hasNext() calls, so multiple consecutive hasNext() calls will actually skip over elements. While this is not necessarily a bug, it is atypical iterator behavior and should be resolved to prevent user error. > KeyValueBlockIterator#nextBlock skips valid blocks > -- > > Key: HDDS-3976 > URL: https://issues.apache.org/jira/browse/HDDS-3976 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Major > > HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced > another one in KeyValueBlockIterator#nextBlock, which depends on the behavior > of that method. When the first key encountered does not pass the filter, the > internal nextBlock field is never intialized. Then a call to nextBlock() > results in call to hasNext() which returns true, which recursively calls > nextBlock(), again calling hasNext(), etc until the end of the set is reached > and an exception is thrown. This skips all valid keys that may occur past the > first invalid key. > This bug was identified while working on HDDS-3869, which adds a strong > typing layer before objects are serialized into RocksDB for datanode. Due to > RocksDB internals, this changes the database layout so that all prefixed keys > are returned at the beginning of the key set, instead of in the end. Since > the original layout returned all prefixed keys at the end of the key set, > this bug was not evident in any of the original unit tests, since the > behavior described above could not occur. > Additionally, the current implementation modifies the internal iterator's > state on hasNext() calls, so multiple consecutive hasNext() calls will > actually skip over elements. While this is not necessarily a bug, it is > atypical iterator behavior and should be resolved to prevent user error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3976) KeyValueBlockIterator#nextBlock skips valid blocks
Ethan Rose created HDDS-3976: Summary: KeyValueBlockIterator#nextBlock skips valid blocks Key: HDDS-3976 URL: https://issues.apache.org/jira/browse/HDDS-3976 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Ethan Rose Assignee: Ethan Rose HDDS-3854 fixed a bug in KeyValueBlockIterator#hasNext, but introduced another one in KeyValueBlockIterator#nextBlock, which depends on the behavior of that method. When the first key encountered does not pass the filter, the internal nextBlock field is never intialized. Then a call to nextBlock() results in call to hasNext() which returns true, which recursively calls nextBlock() etc until the end of the set is reached and an exception is thrown. This skips all valid keys that may occur past the first invalid key. This bug was identified while working on HDDS-3869, which adds a strong typing layer before objects are serialized into RocksDB for datanode. Due to RocksDB internals, this changes the database layout so that all prefixed keys are returned at the beginning of the key set, instead of in the end. Since the original layout returned all prefixed keys at the end of the key set, this bug was not evident in any of the original unit tests, since the behavior described above could not occur. Additionally, the current implementation modifies the internal iterator's state on hasNext() calls, so multiple consecutive hasNext() calls will actually skip over elements. While this is not necessarily a bug, it is atypical iterator behavior and should be resolved to prevent user error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx removed a comment on pull request #1210: HDDS-3965. SCM failed to start up for duplicated pipeline detected.
avijayanhwx removed a comment on pull request #1210:
URL: https://github.com/apache/hadoop-ozone/pull/1210#issuecomment-659593212
First Restart with HDDS-3925.
`2020-07-16 18:22:21,133 [main] INFO pipeline.SCMPipelineManager: Processing
pipeline PipelineID=096c944b-07d9-4a5b-af11-1102b46cf457
2020-07-16 18:22:21,133 [main] INFO pipeline.SCMPipelineManager: Found
pipeline in old format key : PipelineID=096c944b-07d9-4a5b-af11-1102b46cf457
2020-07-16 18:22:21,133 [main] INFO db.RDBStoreIterator: Deleting pipelineID
from DB : 117e0edf-ae20-4d43-8f8f-68edab965925
2020-07-16 18:22:21,141 [main] INFO pipeline.PipelineStateManager: Created
pipeline Pipeline[ Id: 096c944b-07d9-4a5b-af11-1102b46cf457, Nodes:
690d95ba-9af3-41fe-a4e8-fd635072d724{ip: 172.24.0.7, host:
ozone_datanode_3.ozone_default, networkLocation: /default-rack, certSerialId:
null}, Type:RATIS, Factor:ONE, State:ALLOCATED, leaderId:,
CreationTimestamp2020-07-16T18:22:21.132927Z]
2020-07-16 18:22:21,142 [main] INFO pipeline.SCMPipelineManager: Processing
pipeline PipelineID=117e0edf-ae20-4d43-8f8f-68edab965925
2020-07-16 18:22:21,142 [main] INFO pipeline.SCMPipelineManager: Found
pipeline in old format key : PipelineID=117e0edf-ae20-4d43-8f8f-68edab965925
2020-07-16 18:22:21,143 [main] INFO db.RDBStoreIterator: Deleting pipelineID
from DB : 33950078-664e-44f5-b8e8-a53120a0ace4
2020-07-16 18:22:21,143 [main] INFO pipeline.PipelineStateManager: Created
pipeline Pipeline[ Id: 117e0edf-ae20-4d43-8f8f-68edab965925, Nodes:
d3827311-dda3-40a3-9a8c-10ab2d9bcc66{ip: 172.24.0.6, host:
ozone_datanode_1.ozone_default, networkLocation: /default-rack, certSerialId:
null}, Type:RATIS, Factor:ONE, State:ALLOCATED, leaderId:,
CreationTimestamp2020-07-16T18:22:21.142554Z]
2020-07-16 18:22:21,144 [main] INFO pipeline.SCMPipelineManager: Processing
pipeline PipelineID=33950078-664e-44f5-b8e8-a53120a0ace4
2020-07-16 18:22:21,144 [main] INFO pipeline.SCMPipelineManager: Found
pipeline in old format key : PipelineID=33950078-664e-44f5-b8e8-a53120a0ace4
2020-07-16 18:22:21,144 [main] INFO db.RDBStoreIterator: Deleting pipelineID
from DB : 5ddf942e-ccbe-4482-a3b4-49fa6af1f34c
2020-07-16 18:22:21,145 [main] INFO pipeline.PipelineStateManager: Created
pipeline Pipeline[ Id: 33950078-664e-44f5-b8e8-a53120a0ace4, Nodes:
690d95ba-9af3-41fe-a4e8-fd635072d724{ip: 172.24.0.7, host:
ozone_datanode_3.ozone_default, networkLocation: /default-rack, certSerialId:
null}d3827311-dda3-40a3-9a8c-10ab2d9bcc66{ip: 172.24.0.6, host:
ozone_datanode_1.ozone_default, networkLocation: /default-rack, certSerialId:
null}1e6d6f78-d8e3-4e31-9408-8eecf44d3d90{ip: 172.24.0.2, host:
ozone_datanode_2.ozone_default, networkLocation: /default-rack, certSerialId:
null}, Type:RATIS, Factor:THREE, State:ALLOCATED, leaderId:,
CreationTimestamp2020-07-16T18:22:21.144300Z]
2020-07-16 18:22:21,146 [main] INFO pipeline.SCMPipelineManager: Processing
pipeline PipelineID=5ddf942e-ccbe-4482-a3b4-49fa6af1f34c
2020-07-16 18:22:21,146 [main] INFO pipeline.SCMPipelineManager: Found
pipeline in old format key : PipelineID=5ddf942e-ccbe-4482-a3b4-49fa6af1f34c
2020-07-16 18:22:21,146 [main] INFO db.RDBStoreIterator: Unable to delete
key as it does not exist.`
Second Restart with HDDS-3925
`2020-07-16 18:22:54,026 [main] ERROR server.StorageContainerManagerStarter:
SCM start failed with exception
java.io.IOException: Duplicate pipeline ID
PipelineID=096c944b-07d9-4a5b-af11-1102b46cf457 detected.
at
org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.addPipeline(PipelineStateMap.java:86)
at
org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.addPipeline(PipelineStateManager.java:53)
at
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.initializePipelineState(SCMPipelineManager.java:165)
at
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.(SCMPipelineManager.java:100)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.initializeSystemManagers(StorageContainerManager.java:410)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:281)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:213)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:624)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:144)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:119)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:67)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:43)
at picocli.CommandLine.execute(CommandLine.jav
[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1210: HDDS-3965. SCM failed to start up for duplicated pipeline detected.
avijayanhwx commented on pull request #1210:
URL: https://github.com/apache/hadoop-ozone/pull/1210#issuecomment-659593212
First Restart with HDDS-3925.
`2020-07-16 18:22:21,133 [main] INFO pipeline.SCMPipelineManager: Processing
pipeline PipelineID=096c944b-07d9-4a5b-af11-1102b46cf457
2020-07-16 18:22:21,133 [main] INFO pipeline.SCMPipelineManager: Found
pipeline in old format key : PipelineID=096c944b-07d9-4a5b-af11-1102b46cf457
2020-07-16 18:22:21,133 [main] INFO db.RDBStoreIterator: Deleting pipelineID
from DB : 117e0edf-ae20-4d43-8f8f-68edab965925
2020-07-16 18:22:21,141 [main] INFO pipeline.PipelineStateManager: Created
pipeline Pipeline[ Id: 096c944b-07d9-4a5b-af11-1102b46cf457, Nodes:
690d95ba-9af3-41fe-a4e8-fd635072d724{ip: 172.24.0.7, host:
ozone_datanode_3.ozone_default, networkLocation: /default-rack, certSerialId:
null}, Type:RATIS, Factor:ONE, State:ALLOCATED, leaderId:,
CreationTimestamp2020-07-16T18:22:21.132927Z]
2020-07-16 18:22:21,142 [main] INFO pipeline.SCMPipelineManager: Processing
pipeline PipelineID=117e0edf-ae20-4d43-8f8f-68edab965925
2020-07-16 18:22:21,142 [main] INFO pipeline.SCMPipelineManager: Found
pipeline in old format key : PipelineID=117e0edf-ae20-4d43-8f8f-68edab965925
2020-07-16 18:22:21,143 [main] INFO db.RDBStoreIterator: Deleting pipelineID
from DB : 33950078-664e-44f5-b8e8-a53120a0ace4
2020-07-16 18:22:21,143 [main] INFO pipeline.PipelineStateManager: Created
pipeline Pipeline[ Id: 117e0edf-ae20-4d43-8f8f-68edab965925, Nodes:
d3827311-dda3-40a3-9a8c-10ab2d9bcc66{ip: 172.24.0.6, host:
ozone_datanode_1.ozone_default, networkLocation: /default-rack, certSerialId:
null}, Type:RATIS, Factor:ONE, State:ALLOCATED, leaderId:,
CreationTimestamp2020-07-16T18:22:21.142554Z]
2020-07-16 18:22:21,144 [main] INFO pipeline.SCMPipelineManager: Processing
pipeline PipelineID=33950078-664e-44f5-b8e8-a53120a0ace4
2020-07-16 18:22:21,144 [main] INFO pipeline.SCMPipelineManager: Found
pipeline in old format key : PipelineID=33950078-664e-44f5-b8e8-a53120a0ace4
2020-07-16 18:22:21,144 [main] INFO db.RDBStoreIterator: Deleting pipelineID
from DB : 5ddf942e-ccbe-4482-a3b4-49fa6af1f34c
2020-07-16 18:22:21,145 [main] INFO pipeline.PipelineStateManager: Created
pipeline Pipeline[ Id: 33950078-664e-44f5-b8e8-a53120a0ace4, Nodes:
690d95ba-9af3-41fe-a4e8-fd635072d724{ip: 172.24.0.7, host:
ozone_datanode_3.ozone_default, networkLocation: /default-rack, certSerialId:
null}d3827311-dda3-40a3-9a8c-10ab2d9bcc66{ip: 172.24.0.6, host:
ozone_datanode_1.ozone_default, networkLocation: /default-rack, certSerialId:
null}1e6d6f78-d8e3-4e31-9408-8eecf44d3d90{ip: 172.24.0.2, host:
ozone_datanode_2.ozone_default, networkLocation: /default-rack, certSerialId:
null}, Type:RATIS, Factor:THREE, State:ALLOCATED, leaderId:,
CreationTimestamp2020-07-16T18:22:21.144300Z]
2020-07-16 18:22:21,146 [main] INFO pipeline.SCMPipelineManager: Processing
pipeline PipelineID=5ddf942e-ccbe-4482-a3b4-49fa6af1f34c
2020-07-16 18:22:21,146 [main] INFO pipeline.SCMPipelineManager: Found
pipeline in old format key : PipelineID=5ddf942e-ccbe-4482-a3b4-49fa6af1f34c
2020-07-16 18:22:21,146 [main] INFO db.RDBStoreIterator: Unable to delete
key as it does not exist.`
Second Restart with HDDS-3925
`2020-07-16 18:22:54,026 [main] ERROR server.StorageContainerManagerStarter:
SCM start failed with exception
java.io.IOException: Duplicate pipeline ID
PipelineID=096c944b-07d9-4a5b-af11-1102b46cf457 detected.
at
org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.addPipeline(PipelineStateMap.java:86)
at
org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.addPipeline(PipelineStateManager.java:53)
at
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.initializePipelineState(SCMPipelineManager.java:165)
at
org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.(SCMPipelineManager.java:100)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.initializeSystemManagers(StorageContainerManager.java:410)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:281)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:213)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:624)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:144)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:119)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:67)
at
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.call(StorageContainerManagerStarter.java:43)
at picocli.CommandLine.execute(CommandLine.java:1173)
[GitHub] [hadoop-ozone] avijayanhwx opened a new pull request #1210: HDDS-3965. SCM failed to start up for duplicated pipeline detected.
avijayanhwx opened a new pull request #1210: URL: https://github.com/apache/hadoop-ozone/pull/1210 ## What changes were proposed in this pull request? After the patch from HDDS-3925, SCM failed to start up for duplicated pipeline detected. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3965 ## How was this patch tested? Manually tested. Adding unit tests (WIP). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3965) SCM failed to start up for duplicated pipeline detected
[
https://issues.apache.org/jira/browse/HDDS-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-3965:
-
Labels: pull-request-available upgrade-p0 (was: upgrade-p0)
> SCM failed to start up for duplicated pipeline detected
> ---
>
> Key: HDDS-3965
> URL: https://issues.apache.org/jira/browse/HDDS-3965
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Assignee: Aravindan Vijayan
>Priority: Blocker
> Labels: pull-request-available, upgrade-p0
>
> SCM LOG：
> {code}
> 2020-07-15 19:25:09,768 [main] ERROR
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter: SCM start
> failed with exception
> java.io.IOException: Duplicate pipeline ID
> PipelineID=db5966ec-140f-48d8-b0d6-e6f2ff777a77 detected.
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.addPipeline(PipelineStateMap.java:89)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.addPipeline(PipelineStateManager.java:53)
> at
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.initializePipelineState(SCMPipelineManager.java:165)
> at
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.(SCMPipelineManager.java:100)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.initializeSystemManagers(StorageContainerManager.java:410)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:281)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:213)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:624)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:144)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:119)
> RocksDB dump, string,
> rocksdb_ldb --db=scm.db scan --column_family=pipelines
> $db5966ec-140f-48d8-b0d6-e6f2ff777a77ؑ٬??? :
> ?
> $02d3c9b4-7972-4471-a520-fff108b8d32e
> 10.73.33.62
> 10.73.33.62"
> RATIS?M"
> /default-rack???Ƕ?Ő *?71-a520-fff108b8d32e:
> $db5966ec-140f-48d8-b0d6-e6f2ff777a77ؑ٬???2
> ?Yf?Hذwzw :
> ?
> $02d3c9b4-7972-4471-a520-fff108b8d32e
> 10.73.33.62
> 10.73.33.62"
> RATIS?M"
> HEX:
> 0x0A2464623539363665632D313430662D343864382D623064362D65366632373737613737A2061608D891BDA0C1DDD9ACDB0110F7F4DDFBAFDEB9EBB001
> :
> 0x0AAA010A2430326433633962342D373937322D343437312D613532302D66313038623864333265120B31302E37332E2E36321A0B31302E37332E2E3632220A0A05524154495310824D220F0A0A5354414E44414C4F4E4510834D322430326433633962342D373937322D343437312D613532302D663130386238643332653A0D2F64656661756C742D7261636BA2061508F188C9CBC7B6F2E90210AEA6E3C590FEBF90A5011001180120012A3F0A2464623539363665632D313430662D343864382D623064362D65366632373737613737A2061608D891BDA0C1DDD9ACDB0110F7F4DDFBAFDEB9EBB00132004085A7C1E5B42E
> 0xDB5966EC140F48D8B0D6E6F2FF777A77 :
> 0x0AAC010A2430326433633962342D373937322D343437312D613532302D66313038623864333265120B31302E37332E2E36321A0B31302E37332E2E3632220A0A05524154495310824D220F0A0A5354414E44414C4F4E4510834D322430326433633962342D373937322D343437312D613532302D663130386238643332653A0D2F64656661756C742D7261636B4800A2061508F188C9CBC7B6F2E90210AEA6E3C590FEBF90A5011001180120012A3F0A2464623539363665632D313430662D343864382D623064362D65366632373737613737A2061608D891BDA0C1DDD9ACDB0110F7F4DDFBAFDEB9EBB0013200409DFCAF8BB52E
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] prashantpogde commented on pull request #1182: HDDS-3926. OM Token Identifier table should use in-house serialization.
prashantpogde commented on pull request #1182: URL: https://github.com/apache/hadoop-ozone/pull/1182#issuecomment-659580090 Made the changes as suggested by @xiaoyuyao . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1182: HDDS-3926. OM Token Identifier table should use in-house serialization.
prashantpogde commented on a change in pull request #1182:
URL: https://github.com/apache/hadoop-ozone/pull/1182#discussion_r455974965
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/codec/TokenIdentifierCodec.java
##
@@ -42,8 +42,9 @@ public OzoneTokenIdentifier fromPersistedFormat(byte[]
rawData)
Preconditions.checkNotNull(rawData,
"Null byte array can't converted to real object.");
try {
- return OzoneTokenIdentifier.readProtoBuf(rawData);
-} catch (InvalidProtocolBufferException e) {
+ OzoneTokenIdentifier object = OzoneTokenIdentifier.newInstance();
+ return object.fromUniqueSerializedKey(rawData);
+} catch (BufferUnderflowException e) {
Review comment:
done
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/codec/TokenIdentifierCodec.java
##
@@ -42,8 +42,9 @@ public OzoneTokenIdentifier fromPersistedFormat(byte[]
rawData)
Preconditions.checkNotNull(rawData,
"Null byte array can't converted to real object.");
try {
- return OzoneTokenIdentifier.readProtoBuf(rawData);
-} catch (InvalidProtocolBufferException e) {
+ OzoneTokenIdentifier object = OzoneTokenIdentifier.newInstance();
+ return object.fromUniqueSerializedKey(rawData);
+} catch (BufferUnderflowException e) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455968767
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##
@@ -79,15 +79,11 @@ public static OMPathInfo verifyFilesInPath(
String dbDirKeyName = omMetadataManager.getOzoneDirKey(volumeName,
bucketName, pathName);
- if (omMetadataManager.getKeyTable().isExist(dbKeyName)) {
-// Found a file in the given path.
-// Check if this is actual file or a file in the given path
-if (dbKeyName.equals(fileNameFromDetails)) {
- result = OMDirectoryResult.FILE_EXISTS;
-} else {
- result = OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
-}
- } else if (omMetadataManager.getKeyTable().isExist(dbDirKeyName)) {
+ // Check first for dir. This is to handle leading "/" in path, which
Review comment:
Yes. This change was required, when we are allowing leading / in
keyName. I will revert the change
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.
bharatviswa504 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455968524
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##
@@ -79,15 +79,11 @@ public static OMPathInfo verifyFilesInPath(
String dbDirKeyName = omMetadataManager.getOzoneDirKey(volumeName,
bucketName, pathName);
- if (omMetadataManager.getKeyTable().isExist(dbKeyName)) {
Review comment:
This change can be reverted as now we don't allow leading / in path
names when OMConfig enable.filesystems is set to true. Will updated in my next
PR update.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455961310
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##
@@ -79,15 +79,11 @@ public static OMPathInfo verifyFilesInPath(
String dbDirKeyName = omMetadataManager.getOzoneDirKey(volumeName,
bucketName, pathName);
- if (omMetadataManager.getKeyTable().isExist(dbKeyName)) {
Review comment:
I am a little bit worried about changes to OMFileRequest. There is a
risk of changing the behavior and invalidating the app-compat testing we have
done so far.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455960385
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##
@@ -79,15 +79,11 @@ public static OMPathInfo verifyFilesInPath(
String dbDirKeyName = omMetadataManager.getOzoneDirKey(volumeName,
bucketName, pathName);
- if (omMetadataManager.getKeyTable().isExist(dbKeyName)) {
-// Found a file in the given path.
-// Check if this is actual file or a file in the given path
-if (dbKeyName.equals(fileNameFromDetails)) {
- result = OMDirectoryResult.FILE_EXISTS;
-} else {
- result = OMDirectoryResult.FILE_EXISTS_IN_GIVENPATH;
-}
- } else if (omMetadataManager.getKeyTable().isExist(dbDirKeyName)) {
+ // Check first for dir. This is to handle leading "/" in path, which
Review comment:
Leading `/` in path should be dropped at this point, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1196: HDDS-3955. Unable to list intermediate paths on keys created using S3G.
arp7 commented on a change in pull request #1196:
URL: https://github.com/apache/hadoop-ozone/pull/1196#discussion_r455955940
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##
@@ -265,4 +272,30 @@ public AuditMessage buildAuditMessage(AuditAction op,
auditMap.put(OzoneConsts.VOLUME, volume);
return auditMap;
}
+
+
+ public static String getNormalizedKey(boolean enableFileSystemPaths,
+ String keyName) {
+if (enableFileSystemPaths) {
+ return getNormalizedKey(keyName);
+} else {
+ return keyName;
+}
+ }
+
+ @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+ public static String getNormalizedKey(String keyName) {
+String normalizedKeyName;
+if (keyName.startsWith(OM_KEY_PREFIX)) {
+ normalizedKeyName = Paths.get(keyName).toUri().normalize().getPath();
+} else {
+ normalizedKeyName = Paths.get(OM_KEY_PREFIX, keyName).toUri()
+ .normalize().getPath();
+}
+if (!keyName.equals(normalizedKeyName)) {
+ LOG.debug("Normalized key {} to {} ", keyName,
Review comment:
Thanks for adding this log message. It will be super useful!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 commented on a change in pull request #1129: HDDS-3741. Reload old OM state if Install Snapshot from Leader fails
arp7 commented on a change in pull request #1129:
URL: https://github.com/apache/hadoop-ozone/pull/1129#discussion_r455950095
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -3041,58 +3047,74 @@ public TermIndex installSnapshot(String leaderId) {
omRatisServer.getOmStateMachine().pause();
} catch (Exception e) {
LOG.error("Failed to stop/ pause the services. Cannot proceed with " +
- "installing the new checkpoint.", e);
- return null;
-}
-
-//TODO: un-pause SM if any failures and retry?
-
-long lastAppliedIndex = omRatisServer.getLastAppliedTermIndex().getIndex();
-
-boolean canProceed =
-OzoneManagerRatisUtils.verifyTransactionInfo(omTransactionInfo,
-lastAppliedIndex, leaderId, newDBlocation);
-
-// If downloaded DB has transaction info less than current one, return.
-if (!canProceed) {
- return null;
+ "installing the new checkpoint.");
+ // During stopServices, if KeyManager was stopped successfully and
+ // OMMetadataManager stop failed, we should restart the KeyManager.
+ keyManager.start(configuration);
+ throw e;
}
-long leaderIndex = omTransactionInfo.getTransactionIndex();
-long leaderTerm = omTransactionInfo.getCurrentTerm();
+File dbBackup = null;
+TermIndex termIndex = omRatisServer.getLastAppliedTermIndex();
+long term = termIndex.getTerm();
+long lastAppliedIndex = termIndex.getIndex();
+// Check if current applied log index is smaller than the downloaded
+// checkpoint transaction index. If yes, proceed by stopping the ratis
+// server so that the OM state can be re-initialized. If no then do not
+// proceed with installSnapshot.
+boolean canProceed = OzoneManagerRatisUtils.verifyTransactionInfo(
+checkpointTrxnInfo, lastAppliedIndex, leaderId, checkpointLocation);
-File dbBackup;
-try {
- dbBackup = replaceOMDBWithCheckpoint(lastAppliedIndex, oldDBLocation,
- newDBlocation);
-} catch (Exception e) {
- LOG.error("OM DB checkpoint replacement with new downloaded checkpoint "
+
- "failed.", e);
- return null;
+if (canProceed) {
+ try {
+dbBackup = replaceOMDBWithCheckpoint(lastAppliedIndex, oldDBLocation,
+checkpointLocation);
+term = checkpointTrxnInfo.getTerm();
+lastAppliedIndex = checkpointTrxnInfo.getTransactionIndex();
+LOG.info("Replaced DB with checkpoint from OM: {}, term: {}, index:
{}",
+leaderId, term, lastAppliedIndex);
+ } catch (Exception e) {
+LOG.error("Failed to install Snapshot from {} as OM failed to replace"
+
+" DB with downloaded checkpoint. Reloading old OM state.", e);
+ }
+} else {
+ LOG.warn("Cannot proceed with InstallSnapshot as OM is at TermIndex {} "
+
+ "and checkpoint has lower TermIndex {}. Reloading old state of OM.",
+ termIndex, checkpointTrxnInfo.getTermIndex());
}
// Reload the OM DB store with the new checkpoint.
// Restart (unpause) the state machine and update its last applied index
// to the installed checkpoint's snapshot index.
try {
- reloadOMState(leaderIndex, leaderTerm);
- omRatisServer.getOmStateMachine().unpause(leaderIndex, leaderTerm);
-} catch (IOException e) {
- LOG.error("Failed to reload OM state with new DB checkpoint.", e);
- return null;
+ reloadOMState(lastAppliedIndex, term);
+ omRatisServer.getOmStateMachine().unpause(lastAppliedIndex, term);
+ LOG.info("Reloaded OM state with Term: {} and Index: {}", term,
+ lastAppliedIndex);
+} catch (IOException ex) {
+ String errorMsg = "Failed to reload OM state and instantiate services.";
+ exitManager.exitSystem(1, errorMsg, ex, LOG);
}
// Delete the backup DB
try {
- FileUtils.deleteFully(dbBackup);
+ if (dbBackup != null) {
+FileUtils.deleteFully(dbBackup);
+ }
Review comment:
Let's do this in a followup jira. Ok to commit this one.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3965) SCM failed to start up for duplicated pipeline detected
[
https://issues.apache.org/jira/browse/HDDS-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle reassigned HDDS-3965:
-
Assignee: Aravindan Vijayan (was: Prashant Pogde)
[~avijayan] Can you take this up since it is blocking the upgrade-p0 Jiras?
> SCM failed to start up for duplicated pipeline detected
> ---
>
> Key: HDDS-3965
> URL: https://issues.apache.org/jira/browse/HDDS-3965
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Assignee: Aravindan Vijayan
>Priority: Blocker
> Labels: upgrade-p0
>
> SCM LOG：
> {code}
> 2020-07-15 19:25:09,768 [main] ERROR
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter: SCM start
> failed with exception
> java.io.IOException: Duplicate pipeline ID
> PipelineID=db5966ec-140f-48d8-b0d6-e6f2ff777a77 detected.
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.addPipeline(PipelineStateMap.java:89)
> at
> org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.addPipeline(PipelineStateManager.java:53)
> at
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.initializePipelineState(SCMPipelineManager.java:165)
> at
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.(SCMPipelineManager.java:100)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.initializeSystemManagers(StorageContainerManager.java:410)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:281)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.(StorageContainerManager.java:213)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManager.createSCM(StorageContainerManager.java:624)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter$SCMStarterHelper.start(StorageContainerManagerStarter.java:144)
> at
> org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter.startScm(StorageContainerManagerStarter.java:119)
> RocksDB dump, string,
> rocksdb_ldb --db=scm.db scan --column_family=pipelines
> $db5966ec-140f-48d8-b0d6-e6f2ff777a77ؑ٬??? :
> ?
> $02d3c9b4-7972-4471-a520-fff108b8d32e
> 10.73.33.62
> 10.73.33.62"
> RATIS?M"
> /default-rack???Ƕ?Ő *?71-a520-fff108b8d32e:
> $db5966ec-140f-48d8-b0d6-e6f2ff777a77ؑ٬???2
> ?Yf?Hذwzw :
> ?
> $02d3c9b4-7972-4471-a520-fff108b8d32e
> 10.73.33.62
> 10.73.33.62"
> RATIS?M"
> HEX:
> 0x0A2464623539363665632D313430662D343864382D623064362D65366632373737613737A2061608D891BDA0C1DDD9ACDB0110F7F4DDFBAFDEB9EBB001
> :
> 0x0AAA010A2430326433633962342D373937322D343437312D613532302D66313038623864333265120B31302E37332E2E36321A0B31302E37332E2E3632220A0A05524154495310824D220F0A0A5354414E44414C4F4E4510834D322430326433633962342D373937322D343437312D613532302D663130386238643332653A0D2F64656661756C742D7261636BA2061508F188C9CBC7B6F2E90210AEA6E3C590FEBF90A5011001180120012A3F0A2464623539363665632D313430662D343864382D623064362D65366632373737613737A2061608D891BDA0C1DDD9ACDB0110F7F4DDFBAFDEB9EBB00132004085A7C1E5B42E
> 0xDB5966EC140F48D8B0D6E6F2FF777A77 :
> 0x0AAC010A2430326433633962342D373937322D343437312D613532302D66313038623864333265120B31302E37332E2E36321A0B31302E37332E2E3632220A0A05524154495310824D220F0A0A5354414E44414C4F4E4510834D322430326433633962342D373937322D343437312D613532302D663130386238643332653A0D2F64656661756C742D7261636B4800A2061508F188C9CBC7B6F2E90210AEA6E3C590FEBF90A5011001180120012A3F0A2464623539363665632D313430662D343864382D623064362D65366632373737613737A2061608D891BDA0C1DDD9ACDB0110F7F4DDFBAFDEB9EBB0013200409DFCAF8BB52E
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3877) Do not fail CI check for log upload failure
[ https://issues.apache.org/jira/browse/HDDS-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3877: - Labels: pull-request-available (was: ) > Do not fail CI check for log upload failure > --- > > Key: HDDS-3877 > URL: https://issues.apache.org/jira/browse/HDDS-3877 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: CI >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2020-06-26 at 13.08.30.png > > > This workflow failed only due to temporary problem during artifact upload > after successful tests: https://github.com/apache/hadoop-ozone/runs/809777550 > To save time and resources, checks should not fail in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #1209: HDDS-3877. Do not fail CI check for log upload failure
adoroszlai opened a new pull request #1209: URL: https://github.com/apache/hadoop-ozone/pull/1209 ## What changes were proposed in this pull request? Consider CI successful even if `upload-artifact` step fails, to save time and resources. Examples where this happened: https://github.com/apache/hadoop-ozone/runs/809777550 https://github.com/adoroszlai/hadoop-ozone/runs/878300331 https://issues.apache.org/jira/browse/HDDS-3877 ## How was this patch tested? Regular CI (to validate syntax): https://github.com/adoroszlai/hadoop-ozone/runs/815643051 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1204: HDDS-3964. Ratis config key mismatch
adoroszlai commented on pull request #1204: URL: https://github.com/apache/hadoop-ozone/pull/1204#issuecomment-659519673 > > I was considering changing the parameter to `Duration` > yeah...that seems like a better approach. Created HDDS-3975 for this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1104: HDDS-3612. Allow mounting bucket under other volume
adoroszlai commented on pull request #1104: URL: https://github.com/apache/hadoop-ozone/pull/1104#issuecomment-659518247 > I take back this comment. This feature helps to expose buckets created via shell/ofs to S3. So making it configurable is not needed. (As by default if user decides to expose any bucket created via shell, it can be exposed without any restart.) > > Thank You @arp7 for the offline discussion. OK. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1104: HDDS-3612. Allow mounting bucket under other volume
adoroszlai commented on pull request #1104: URL: https://github.com/apache/hadoop-ozone/pull/1104#issuecomment-659517803 > I have one suggestion, can we make this mount configurable option, as this is required only when old S3 buckets needs to be exposed via O3fs/ofs. In this way, we can avoid extra bucket checks. (I know that it is in-memory cache, but still I feel if some one does not require this feature, it is totally disabled). I think we can make that check in 2 places. One in create bucket/and other in resolveBucketLink. > > Fine to do in another Jira also. Thanks for the suggestion, sounds good to me. I think if we want to disable it by default, then it should be done as part of this task. If it's enabled by default, then it's fine to add the config later in separate task. Let me know what should be the default value. CC @arp7 @elek This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1104: HDDS-3612. Allow mounting bucket under other volume
bharatviswa504 commented on pull request #1104: URL: https://github.com/apache/hadoop-ozone/pull/1104#issuecomment-659515813 > I have one suggestion, can we make this mount configurable option, as this is required only when old S3 buckets needs to be exposed via O3fs/ofs. In this way, we can avoid extra bucket checks. (I know that it is in-memory cache, but still I feel if some one does not require this feature, it is totally disabled). I think we can make that check in 2 places. One in create bucket/and other in resolveBucketLink. > > Fine to do in another Jira also. I take back this comment. This feature helps to expose buckets created via shell/ofs to S3. So making it configurable is not needed. (As by default if user decides to expose any bucket created via shell, it can be exposed without any restart.) Thank You @arp7 for the offline discussion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1104: HDDS-3612. Allow mounting bucket under other volume
bharatviswa504 commented on pull request #1104: URL: https://github.com/apache/hadoop-ozone/pull/1104#issuecomment-659503325 I have one suggestion, can we make this mount configurable option, as this is required only when old S3 buckets needs to be exposed via O3fs/ofs. In this way, we can avoid extra bucket checks. (I know that it is in-memory cache, but still I feel if some one does not require this feature, it is totally disabled). I think we can make that check in 2 places. One in create bucket/and other in resolveBucketLink. Fine to do in another Jira also. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1104: HDDS-3612. Allow mounting bucket under other volume
bharatviswa504 commented on pull request #1104: URL: https://github.com/apache/hadoop-ozone/pull/1104#issuecomment-659501553 > > > > When deleting a bucket, we check bucket is empty or not. Do we need to check resolvedBucket is empty or not? > > > > > > > > > Link can be deleted regardless of the existence or emptiness of the real bucket it points to. So no need to check resolved bucket. > > > > > > This is the semantics for link buckets, because this is not a delete link, it is same delete bucket op which OM has received. > > The same "delete bucket" op works for deleting links. Since links cannot have direct entries, the emptiness check will not find any, hence delete is always allowed. We don't resolve the bucket the link points to, because its emptiness does not affect whether we can delete the link. > > Please let me know if I misunderstood the question. Makes sense to me. Thanks for the explanation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-698) Support Topology Awareness for Ozone
[ https://issues.apache.org/jira/browse/HDDS-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159256#comment-17159256 ] Stephen O'Donnell commented on HDDS-698: [~Sammi] I think we can close this overall Jira now. Any further topology issues can be handled as normal Jiras without this umbrella Jira. > Support Topology Awareness for Ozone > > > Key: HDDS-698 > URL: https://issues.apache.org/jira/browse/HDDS-698 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: SCM >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Blocker > Fix For: 0.6.0 > > Attachments: HDDS-698.000.patch, network-topology-default.xml, > network-topology-nodegroup.xml > > > This is an umbrella JIRA to add topology aware support for Ozone Pipelines, > Containers and Blocks. Long time since HDFS is created, we provide > rack/nodegroup awareness for reliability and high performance for data > access. Ozone need a similar mechanism and this can be more flexible for > cloud scenarios. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1930) Test Topology Aware Job scheduling with Ozone Topology
[ https://issues.apache.org/jira/browse/HDDS-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159255#comment-17159255 ] Stephen O'Donnell commented on HDDS-1930: - I believe this is not an issue, or no longer an issue if it previously was an issue. In the counters I posted above, we can see there are data-local and rack-local tasks being scheduled. I also enabled debug logging for the application master and you could see if making container allocation decisions based on the locality information. Therefore I am going to close this issue. > Test Topology Aware Job scheduling with Ozone Topology > -- > > Key: HDDS-1930 > URL: https://issues.apache.org/jira/browse/HDDS-1930 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Stephen O'Donnell >Priority: Major > Fix For: 0.6.0 > > > My initial results with Terasort does not seem to report the counter > properly. Most of the requests are handled by rack local but no node local. > This ticket is opened to add more system testing to validate the feature. > Total Allocated Containers: 3778 > Each table cell represents the number of NodeLocal/RackLocal/OffSwitch > containers satisfied by NodeLocal/RackLocal/OffSwitch resource requests. > Node Local RequestRack Local Request Off Switch Request > Num Node Local Containers (satisfied by) 0 > Num Rack Local Containers (satisfied by) 0 3648 > Num Off Switch Containers (satisfied by) 0 96 34 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1930) Test Topology Aware Job scheduling with Ozone Topology
[ https://issues.apache.org/jira/browse/HDDS-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-1930. - Resolution: Not A Problem > Test Topology Aware Job scheduling with Ozone Topology > -- > > Key: HDDS-1930 > URL: https://issues.apache.org/jira/browse/HDDS-1930 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Stephen O'Donnell >Priority: Major > Fix For: 0.6.0 > > > My initial results with Terasort does not seem to report the counter > properly. Most of the requests are handled by rack local but no node local. > This ticket is opened to add more system testing to validate the feature. > Total Allocated Containers: 3778 > Each table cell represents the number of NodeLocal/RackLocal/OffSwitch > containers satisfied by NodeLocal/RackLocal/OffSwitch resource requests. > Node Local RequestRack Local Request Off Switch Request > Num Node Local Containers (satisfied by) 0 > Num Rack Local Containers (satisfied by) 0 3648 > Num Off Switch Containers (satisfied by) 0 96 34 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3975) Use Duration for time in RatisClientConfig
Attila Doroszlai created HDDS-3975:
--
Summary: Use Duration for time in RatisClientConfig
Key: HDDS-3975
URL: https://issues.apache.org/jira/browse/HDDS-3975
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
Change parameter and return type of time-related config methods in
{{RatisClientConfig}} to {{Duration}}. This results in more readable parameter
values and type safety.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3907) Topology related acceptance test is flaky
[
https://issues.apache.org/jira/browse/HDDS-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159235#comment-17159235
]
Sammi Chen commented on HDDS-3907:
--
[~elek], do you think we should fix this in 0.6.0?
> Topology related acceptance test is flaky
> -
>
> Key: HDDS-3907
> URL: https://issues.apache.org/jira/browse/HDDS-3907
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Marton Elek
>Assignee: Xiaoyu Yao
>Priority: Blocker
>
> Examples:
> https://github.com/elek/ozone-build-results/tree/master/2020/06/30/1318/acceptance
> https://github.com/elek/ozone-build-results/tree/master/2020/06/30/1321/acceptance
> https://github.com/elek/ozone-build-results/tree/master/2020/06/30/1334/acceptance
> Some strange errors:
> {code}
> scm_1 | 2020-06-30 19:17:50,787 [RatisPipelineUtilsThread] ERROR
> pipeline.SCMPipelineManager: Failed to create pipeline of type RATIS and
> factor ONE. Exception: Cannot create pipeline of factor 1 using 0 nodes. Used
> 6 nodes. Healthy nodes 6
> scm_1 | 2020-06-30 19:17:50,788 [RatisPipelineUtilsThread] ERROR
> pipeline.SCMPipelineManager: Failed to create pipeline of type RATIS and
> factor THREE. Exception: Pipeline creation failed because nodes are engaged
> in other pipelines and every node can only be engaged in max 2 pipelines.
> Required 3. Found 0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1208: HDDS-3755. Storage-class support for Ozone
elek commented on pull request #1208: URL: https://github.com/apache/hadoop-ozone/pull/1208#issuecomment-659404852 > Would you like to merge apache#920 to make it possible to accept any number of replication which specified by StorageClass or rebase that works to this storageClass POC? Both are ok for me. I think the easiest approach can be to modify only the `ClosedStateConfiguration.replicationFactor` to be an integer. And keep the `OpenStateConfiguration` as is. OpenStateConfiguration can be more tricky, but modifying the closed state seems to be safe. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3755) Storage-class support for Ozone
[ https://issues.apache.org/jira/browse/HDDS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3755: - Labels: pull-request-available (was: ) > Storage-class support for Ozone > --- > > Key: HDDS-3755 > URL: https://issues.apache.org/jira/browse/HDDS-3755 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > > Use a storage-class as an abstraction which combines replication > configuration, container states and transitions. > See this thread for the detailed design doc: > > [https://lists.apache.org/thread.html/r1e2a5d5581abe9dd09834305ca65a6807f37bd229a07b8b31bda32ad%40%3Cozone-dev.hadoop.apache.org%3E] > which is also uploaded to here: > https://hackmd.io/4kxufJBOQNaKn7PKFK_6OQ?edit -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek opened a new pull request #1208: HDDS-3755. Storage-class support for Ozone
elek opened a new pull request #1208: URL: https://github.com/apache/hadoop-ozone/pull/1208 Created together with @maobaolong (thanks the help) ## What changes were proposed in this pull request? This is the initial draft for storage-class support. It introduces the storage-class and uses it instead of replication factor / type. Legacy clients are supported with converting back to factor / type to storage-class. Storage classes are hard-coded in the java code (can be changed later to be configurable). ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3755 ## How was this patch tested? With CI tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on pull request #1204: HDDS-3964. Ratis config key mismatch
lokeshj1703 commented on pull request #1204: URL: https://github.com/apache/hadoop-ozone/pull/1204#issuecomment-659388287 > I was considering changing the parameter to `Duration`, similar to > > https://github.com/apache/hadoop-ozone/blob/de027855798bf3b891b8d3c00dc8e59531f98781/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ReconTaskConfig.java#L41-L49 > > What do you think about that? yeah...that seems like a better approach. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #1207: HDDS-3973. Update main feature design status.
elek commented on a change in pull request #1207: URL: https://github.com/apache/hadoop-ozone/pull/1207#discussion_r455754639 ## File path: hadoop-hdds/docs/content/design/decommissioning.md ## @@ -3,7 +3,7 @@ title: Decommissioning in Ozone summary: Formal process to shut down machines in a safe way after the required replications. date: 2019-07-31 jira: HDDS-1881 -status: implementing +status: implemented Review comment: It's more like abandoned. The 90 % of the features are implemented but they are not merged back to the master and it's not actively rebased. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] github-actions[bot] closed pull request #917: HDDS-3002. Make the Mountd work for Ozone.
github-actions[bot] closed pull request #917: URL: https://github.com/apache/hadoop-ozone/pull/917 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] github-actions[bot] commented on pull request #917: HDDS-3002. Make the Mountd work for Ozone.
github-actions[bot] commented on pull request #917: URL: https://github.com/apache/hadoop-ozone/pull/917#issuecomment-659381579 Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author. It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time. It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs. If you need ANY help to finish this PR, please [contact the community](https://github.com/apache/hadoop-ozone#contact) on the mailing list or the slack channel. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #917: HDDS-3002. Make the Mountd work for Ozone.
elek commented on pull request #917: URL: https://github.com/apache/hadoop-ozone/pull/917#issuecomment-659381363 /close This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1204: HDDS-3964. Ratis config key mismatch
adoroszlai commented on pull request #1204: URL: https://github.com/apache/hadoop-ozone/pull/1204#issuecomment-659376896 Thanks @lokeshj1703 for the review. > Minor comment - We can add suffix like "InMs" for time based configs in classes like RatisClientConfig. > We can do it as part of separate jira though. I was considering changing the parameter to `Duration`, similar to https://github.com/apache/hadoop-ozone/blob/de027855798bf3b891b8d3c00dc8e59531f98781/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ReconTaskConfig.java#L41-L49 What do you think about that? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3964) Ratis config key mismatch
[
https://issues.apache.org/jira/browse/HDDS-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-3964:
---
Target Version/s: 0.6.0 (was: 0.7.0)
> Ratis config key mismatch
> -
>
> Key: HDDS-3964
> URL: https://issues.apache.org/jira/browse/HDDS-3964
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Neo Yang
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
>
> Some of the Ratis configurations in integration tests are not applied due to
> mismatch in config keys.
> #
> [Ratis|https://github.com/apache/incubator-ratis/blob/master/ratis-client/src/main/java/org/apache/ratis/client/RaftClientConfigKeys.java#L41-L53]:
> {{raft.client.rpc.watch.request.timeout}}
>
> [Ozone|https://github.com/apache/hadoop-ozone/blob/926048403d115ddcb59ff130e5c46e518874b8aa/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java#L119-L122]:
> {{raft.client.watch.request.timeout}}
> #
> [Ratis|https://github.com/apache/incubator-ratis/blob/4db4f804aa90f9900cda08c79b54a45f80f4213b/ratis-server/src/main/java/org/apache/ratis/server/RaftServerConfigKeys.java#L470-L473]:
> {{raft.server.notification.no-leader.timeout}}
>
> [Ozone|https://github.com/apache/hadoop-ozone/blob/926048403d115ddcb59ff130e5c46e518874b8aa/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisServerConfig.java#L42]:
> {{raft.server.Notification.no-leader.timeout}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3964) Ratis config key mismatch
[
https://issues.apache.org/jira/browse/HDDS-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-3964:
---
Status: Patch Available (was: In Progress)
> Ratis config key mismatch
> -
>
> Key: HDDS-3964
> URL: https://issues.apache.org/jira/browse/HDDS-3964
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Neo Yang
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
>
> Some of the Ratis configurations in integration tests are not applied due to
> mismatch in config keys.
> #
> [Ratis|https://github.com/apache/incubator-ratis/blob/master/ratis-client/src/main/java/org/apache/ratis/client/RaftClientConfigKeys.java#L41-L53]:
> {{raft.client.rpc.watch.request.timeout}}
>
> [Ozone|https://github.com/apache/hadoop-ozone/blob/926048403d115ddcb59ff130e5c46e518874b8aa/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java#L119-L122]:
> {{raft.client.watch.request.timeout}}
> #
> [Ratis|https://github.com/apache/incubator-ratis/blob/4db4f804aa90f9900cda08c79b54a45f80f4213b/ratis-server/src/main/java/org/apache/ratis/server/RaftServerConfigKeys.java#L470-L473]:
> {{raft.server.notification.no-leader.timeout}}
>
> [Ozone|https://github.com/apache/hadoop-ozone/blob/926048403d115ddcb59ff130e5c46e518874b8aa/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/DatanodeRatisServerConfig.java#L42]:
> {{raft.server.Notification.no-leader.timeout}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1142: HDDS-3855. Add upgrade smoketest
adoroszlai commented on pull request #1142:
URL: https://github.com/apache/hadoop-ozone/pull/1142#issuecomment-659356744
> Overall it looks good to me, and really impressive approach. I have a few
comments -- none of them are blocker, but i like to discuss technical details...
Thanks for taking a look. I waited with the merge exactly to have this kind
of discussion. ;)
> 1. Can you please help me to understand why did you remove `-f
"${compose_file}"`?
Each `-f` accepts only a single filename, so using the same command with one
or more files is easier with `COMPOSE_FILE` approach. Initially I used two
separate files (including the one from `ozone` env), so I needed this fix, but
then abandoned that approach. This part of the change could be extracted to a
separate issue if you prefer to simplify this one a bit. (It allows you to run
`ozone/test.sh` with monitoring enabled, so I'd rather not drop it completely.)
> 2. fixed ip / dedicated network in docker-compose file seems to be
unnecessary in this cluster (IMHO)
> 4. you create external volume directories but `/data` is already a volume
inside the docker containers. If you use simple `docker-compose stop` instead
of `down` it can be reused. Did you consider using this approach?
>
> Why do you prefer external volumes? (I found two arguments: easier to
debug + easier to execute commands when the cluster is down. But interested if
you had any other motivations...).
After `stop/start` this is what `ozone version` prints:
```
//
//
/ /
/ ///
/
/
/ //
/// /
/ ///
/ // /
// // /
///
//
/// //
/0.5.0-beta(Crater Lake)
Source code repository g...@github.com:apache/hadoop-ozone.git -r
9b4f8fd49fa15946994bccc6c6ac50a560cfb0ea
Compiled by dchitlangia on 2020-03-16T00:54Z
Compiled with protoc 2.5.0
From source with checksum 4cde4c7a7aaa250bfbaf58220cb8e2c
Using HDDS 0.5.0-beta
Source code repository g...@github.com:apache/hadoop-ozone.git -r
9b4f8fd49fa15946994bccc6c6ac50a560cfb0ea
Compiled by dchitlangia on 2020-03-16T00:53Z
Compiled with protoc 2.5.0
From source with checksum 9df32efd56424ab869a0acd0124e4bf5
```
So `docker-compose down/up` is needed because changes to the compose file
(docker image, etc.) are not picked up with `stop/start`. And we need different
images before/after upgrade.
That's the reason for both volumes and network settings. I had started out
without the network/ip settings, but the containers did not always get the same
address after `down`/`up`, nor would they reuse volumes.
> 3. It seems to be a big restriction that we can't start multiple datanode
on the same file system without configuring the datanode path. This is the
reason why you need dn1..dn3 directories. I am wondering if we can provide a
generic solution to this one. Maybe we can support `${env...}` notion when we
set the datanode directory?
Would be nice, I think we can explore it later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3974) StateContext::addPipelineActionIfAbsent does not work as expected.
Glen Geng created HDDS-3974:
---
Summary: StateContext::addPipelineActionIfAbsent does not work as
expected.
Key: HDDS-3974
URL: https://issues.apache.org/jira/browse/HDDS-3974
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Datanode
Affects Versions: 0.6.0
Reporter: Glen Geng
Assignee: Aravindan Vijayan
{code:java}
/**
* Add PipelineAction to PipelineAction queue if it's not present.
*
* @param pipelineAction PipelineAction to be added
*/
public void addPipelineActionIfAbsent(PipelineAction pipelineAction) {
synchronized (pipelineActions) {
/**
* If pipelineAction queue already contains entry for the pipeline id
* with same action, we should just return.
* Note: We should not use pipelineActions.contains(pipelineAction) here
* as, pipelineAction has a msg string. So even if two msgs differ though
* action remains same on the given pipeline, it will end up adding it
* multiple times here.
*/
for (InetSocketAddress endpoint : endpoints) {
Queue actionsForEndpoint =
this.pipelineActions.get(endpoint);
for (PipelineAction pipelineActionIter : actionsForEndpoint) {
if (pipelineActionIter.getAction() == pipelineAction.getAction()
&& pipelineActionIter.hasClosePipeline() && pipelineAction
.hasClosePipeline()
&& pipelineActionIter.getClosePipeline().getPipelineID()
.equals(pipelineAction.getClosePipeline().getPipelineID())) {
break;
}
}
actionsForEndpoint.add(pipelineAction);
}
}
}
{code}
no matter absent or not, pipeline action will be added.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3807) Propagate raft log disks info to SCM from datanode
[ https://issues.apache.org/jira/browse/HDDS-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-3807. --- Fix Version/s: 0.6.0 Resolution: Fixed > Propagate raft log disks info to SCM from datanode > -- > > Key: HDDS-3807 > URL: https://issues.apache.org/jira/browse/HDDS-3807 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode, SCM >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > No of pipelines to be created on a datanode is to be driven by the no of raft > log disks configured on datanode. The Jira here is to add support for > propagation of raft log volume info to SCM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek merged pull request #1107: HDDS-3807. Propagate raft log disks info to SCM from datanode.
elek merged pull request #1107: URL: https://github.com/apache/hadoop-ozone/pull/1107 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1107: HDDS-3807. Propagate raft log disks info to SCM from datanode.
elek commented on pull request #1107: URL: https://github.com/apache/hadoop-ozone/pull/1107#issuecomment-659327881 Merging it as checkstyle issues are fixed. Thanks @bshashikant the contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek edited a comment on pull request #1107: HDDS-3807. Propagate raft log disks info to SCM from datanode.
elek edited a comment on pull request #1107: URL: https://github.com/apache/hadoop-ozone/pull/1107#issuecomment-651678810 Checkstyle violation seems to be related. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3967) Remove leftover debug setting
[ https://issues.apache.org/jira/browse/HDDS-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3967: - Fix Version/s: (was: 0.7.0) 0.6.0 > Remove leftover debug setting > - > > Key: HDDS-3967 > URL: https://issues.apache.org/jira/browse/HDDS-3967 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: test >Affects Versions: 0.6.0 >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Minor > Labels: pull-request-available > Fix For: 0.6.0 > > > HDDS-3821 > [accidentally|https://github.com/apache/hadoop-ozone/pull/1101#issuecomment-658750232] > set some [log level to > DEBUG|https://github.com/apache/hadoop-ozone/blob/926048403d115ddcb59ff130e5c46e518874b8aa/hadoop-ozone/integration-test/src/test/resources/log4j.properties#L23-L24] > for integration tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3968) LDB scan fails to read from transactionInfoTable
[ https://issues.apache.org/jira/browse/HDDS-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen resolved HDDS-3968. -- Resolution: Fixed > LDB scan fails to read from transactionInfoTable > > > Key: HDDS-3968 > URL: https://issues.apache.org/jira/browse/HDDS-3968 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > *[root@bv-cml-1 ~]# ozone debug ldb --db=/var/lib/hadoop-ozone/om/data/om.db/ > scan --column_family=transactionInfoTable > Table with specified name does not exist* > This is because DBDefinition is missing transactionInfo table. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3968) LDB scan fails to read from transactionInfoTable
[ https://issues.apache.org/jira/browse/HDDS-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3968: - Fix Version/s: 0.6.0 > LDB scan fails to read from transactionInfoTable > > > Key: HDDS-3968 > URL: https://issues.apache.org/jira/browse/HDDS-3968 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > *[root@bv-cml-1 ~]# ozone debug ldb --db=/var/lib/hadoop-ozone/om/data/om.db/ > scan --column_family=transactionInfoTable > Table with specified name does not exist* > This is because DBDefinition is missing transactionInfo table. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #983: HDDS-3584. Support write 2 replication
elek commented on pull request #983: URL: https://github.com/apache/hadoop-ozone/pull/983#issuecomment-659297817 /pending more discussion is requested by @arp7 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1028: HDDS-3735. Improve SCM performance with 3.7% by remove unnecessary lock and unlock
elek commented on pull request #1028: URL: https://github.com/apache/hadoop-ozone/pull/1028#issuecomment-659297555 > @elek I think this case can not be avoided by only add read lock Agree. It seems to be necessary. But it means that we can't remove the lock in the ContainerStateMap in this patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1207: HDDS-3973. Update main feature design status.
ChenSammi commented on pull request #1207: URL: https://github.com/apache/hadoop-ozone/pull/1207#issuecomment-659293549 @nandakumar131 Could you take a look at this JIRA, make sure I have your name correctly spelled. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
