[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
linyiqun commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r508242000
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##
@@ -583,4 +588,112 @@ public static OmKeyInfo getOmKeyInfoFromFileTable(boolean
openFileTable,
return dbOmKeyInfo;
}
+ /**
+ * Gets OmKeyInfo if exists for the given key name in the DB.
+ *
+ * @param omMetadataMgr metadata manager
+ * @param volumeNamevolume name
+ * @param bucketNamebucket name
+ * @param keyName key name
+ * @param scmBlockSize scm block size
+ * @return OzoneFileStatus
+ * @throws IOException DB failure
+ */
+ @Nullable
+ public static OzoneFileStatus getOMKeyInfoIfExists(
+ OMMetadataManager omMetadataMgr, String volumeName, String bucketName,
+ String keyName, long scmBlockSize) throws IOException {
+
+Path keyPath = Paths.get(keyName);
+Iterator elements = keyPath.iterator();
+String bucketKey = omMetadataMgr.getBucketKey(volumeName, bucketName);
+OmBucketInfo omBucketInfo =
+omMetadataMgr.getBucketTable().get(bucketKey);
+
+long lastKnownParentId = omBucketInfo.getObjectID();
+OmDirectoryInfo omDirInfo = null;
+while (elements.hasNext()) {
+ String fileName = elements.next().toString();
+
+ // For example, /vol1/buck1/a/b/c/d/e/file1.txt
+ // 1. Do lookup path component on directoryTable starting from bucket
+ // 'buck1' to the leaf node component, which is 'file1.txt'.
+ // 2. If there is no dir exists for the leaf node component 'file1.txt'
+ // then do look it on fileTable.
+ String dbNodeName = omMetadataMgr.getOzonePathKey(
+ lastKnownParentId, fileName);
+ omDirInfo = omMetadataMgr.getDirectoryTable().get(dbNodeName);
+
+ if (omDirInfo != null) {
+lastKnownParentId = omDirInfo.getObjectID();
+ } else if (!elements.hasNext()) {
+// reached last path component. Check file exists for the given path.
+OmKeyInfo omKeyInfo = OMFileRequest.getOmKeyInfoFromFileTable(false,
+omMetadataMgr, dbNodeName, keyName);
+if (omKeyInfo != null) {
+ return new OzoneFileStatus(omKeyInfo, scmBlockSize, false);
+}
+ } else {
+// Missing intermediate directory and just return null;
+// key not found in DB
+return null;
+ }
+}
+
+if (omDirInfo != null) {
+ OmKeyInfo omKeyInfo = getOmKeyInfo(volumeName, bucketName, omDirInfo,
+ keyName);
+ return new OzoneFileStatus(omKeyInfo, scmBlockSize, true);
+}
+
+// key not found in DB
+return null;
+ }
+
+ /**
+ * Prepare OmKeyInfo from OmDirectoryInfo.
+ *
+ * @param volumeName volume name
+ * @param bucketName bucket name
+ * @param dirInfodirectory info
+ * @param keyNameusewr given key name
+ * @return OmKeyInfo object
+ */
+ @NotNull
+ public static OmKeyInfo getOmKeyInfo(String volumeName, String bucketName,
+ OmDirectoryInfo dirInfo, String keyName) {
+
+OmKeyInfo.Builder builder = new OmKeyInfo.Builder();
+builder.setParentObjectID(dirInfo.getParentObjectID());
+builder.setKeyName(keyName);
+builder.setAcls(dirInfo.getAcls());
+builder.addAllMetadata(dirInfo.getMetadata());
+builder.setVolumeName(volumeName);
+builder.setBucketName(bucketName);
+builder.setCreationTime(dirInfo.getCreationTime());
+builder.setModificationTime(dirInfo.getModificationTime());
+builder.setObjectID(dirInfo.getObjectID());
+builder.setUpdateID(dirInfo.getUpdateID());
+builder.setFileName(dirInfo.getName());
+builder.setReplicationType(HddsProtos.ReplicationType.RATIS);
+builder.setReplicationFactor(HddsProtos.ReplicationFactor.ONE);
Review comment:
Okay, let us keep current logic here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4358) Delete : make delete an atomic ops for leaf node(empty directory or file)
Rakesh Radhakrishnan created HDDS-4358: -- Summary: Delete : make delete an atomic ops for leaf node(empty directory or file) Key: HDDS-4358 URL: https://issues.apache.org/jira/browse/HDDS-4358 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rakesh Radhakrishnan Assignee: Rakesh Radhakrishnan This task handles only empty directory and file deletions. Recursive deletes will be handled separately in another Jira task. Here in this jira, we consider only new Ozone FS client talking to new OM server. Later, I will raise separate Jira task to handle compatibilities across different client/server versions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4357) Rename : make rename an atomic ops by updating key path entry in dir/file table
Rakesh Radhakrishnan created HDDS-4357: -- Summary: Rename : make rename an atomic ops by updating key path entry in dir/file table Key: HDDS-4357 URL: https://issues.apache.org/jira/browse/HDDS-4357 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Rakesh Radhakrishnan Assignee: Rakesh Radhakrishnan This task is to handle rename key path request and make it an atomic operation by updating the DirTable or FileTable. Here in this jira, we consider only new Ozone FS client talking to new OM server. Later, I will raise separate Jira task to handle compatibilities across different client/server versions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1498: HDDS-4339. Skip aws signature when it's absent in header.
timmylicheng commented on a change in pull request #1498:
URL: https://github.com/apache/hadoop-ozone/pull/1498#discussion_r508182506
##
File path:
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java
##
@@ -116,6 +116,13 @@ private OzoneClient getClient(OzoneConfiguration config)
throws IOException {
}
}
+ // ONLY validate aws access id when needed.
+ private void validateAccessId(String awsAccessId) throws Exception {
+if (awsAccessId == null || awsAccessId.equals("")) {
+ throw S3_AUTHINFO_CREATION_ERROR;
Review comment:
@bharatviswa504 Right now, it catching all exceptions and just print out
logs. Do we want to throw OS3Exception?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4356) SCM is flooded with useless "Deleting blocks" messages
Wei-Chiu Chuang created HDDS-4356:
-
Summary: SCM is flooded with useless "Deleting blocks" messages
Key: HDDS-4356
URL: https://issues.apache.org/jira/browse/HDDS-4356
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: SCM
Affects Versions: 1.0.0
Reporter: Wei-Chiu Chuang
Testing a 1.0.0 SCM.
I'm seeing these messages flooding SCM log file when a dead DN is detected.
{noformat}
2020-10-19 13:48:19,642 INFO org.apache.hadoop.hdds.scm.node.DeadNodeHandler: A
dead datanode is detected. 9b27c38d-9104-491b-b76b-959dc9dd06a2{ip: 10.12.1.82,
host: rhel12.ozone.local, networkLocation: /default, certSerialId: null}
2020-10-19 13:48:24,894 INFO
org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer: SCM is informed by OM
to delete 1000 blocks
2020-10-19 13:48:24,894 INFO org.apache.hadoop.hdds.scm.block.BlockManagerImpl:
Deleting blocks
2020-10-19 13:48:24,894 INFO org.apache.hadoop.hdds.scm.block.BlockManagerImpl:
Deleting blocks
2020-10-19 13:48:24,894 INFO org.apache.hadoop.hdds.scm.block.BlockManagerImpl:
Deleting blocks
2020-10-19 13:48:24,894 INFO org.apache.hadoop.hdds.scm.block.BlockManagerImpl:
Deleting blocks
2020-10-19 13:48:24,894 INFO org.apache.hadoop.hdds.scm.block.BlockManagerImpl:
Deleting blocks
2020-10-19 13:48:24,894 INFO org.apache.hadoop.hdds.scm.block.BlockManagerImpl:
Deleting blocks
2020-10-19 13:48:24,894 INFO org.apache.hadoop.hdds.scm.block.BlockManagerImpl:
Deleting blocks
2020-10-19 13:48:24,895 INFO org.apache.hadoop.hdds.scm.block.BlockManagerImpl:
Deleting blocks
2020-10-19 13:48:24,895 INFO org.apache.hadoop.hdds.scm.block.BlockManagerImpl:
Deleting blocks
2020-10-19 13:48:24,895 INFO org.apache.hadoop.hdds.scm.block.BlockManagerImpl:
Deleting blocks
{noformat}
SCM deletes 1000 blocks max at a time, and the "Deleting blocks" message
repeats for 1000 times. Worse, it doesn't give any useful information (at the
very least, it should print the block id)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4301) SCM CA certificate does not encode KeyUsage extension properly
[
https://issues.apache.org/jira/browse/HDDS-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao resolved HDDS-4301.
--
Fix Version/s: 1.1.0
Resolution: Fixed
> SCM CA certificate does not encode KeyUsage extension properly
> --
>
> Key: HDDS-4301
> URL: https://issues.apache.org/jira/browse/HDDS-4301
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Security
>Affects Versions: 1.0.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> This could be problematic with strict security provider such as FIPS. The
> default non-FIPS provider such as SunJCE and BC provider work fine though.
> This ticket is opened to fix it.
> {code:java}
> 2020-09-30 12:01:52,962 ERROR
> org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer:
> Unable to initialize CertificateServer.
> org.apache.hadoop.hdds.security.exception.SCMSecurityException:
> java.security.cert.CertificateParsingException: cannot construct KeyUsage:
> java.lang.IllegalArgumentException: illegal object in getInstance:
> com.safelogic.cryptocomply.asn1.DEROctetString
> at
> org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec.getPEMEncodedString(CertificateCodec.java:105)
> at
> org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec.writeCertificate(CertificateCodec.java:182)
> at
> org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer.generateRootCertificate(DefaultCAServer.java:495)
> at
> org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer.generateSelfSignedCA(DefaultCAServer.java:303)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on pull request #1468: HDDS-4301. SCM CA certificate does not encode KeyUsage extension properly
xiaoyuyao commented on pull request #1468: URL: https://github.com/apache/hadoop-ozone/pull/1468#issuecomment-712413712 Thanks @elek and @arp7 for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao merged pull request #1468: HDDS-4301. SCM CA certificate does not encode KeyUsage extension properly
xiaoyuyao merged pull request #1468: URL: https://github.com/apache/hadoop-ozone/pull/1468 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4348) Performance evaluation for Ozone trash emptier
[ https://issues.apache.org/jira/browse/HDDS-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian updated HDDS-4348: - Description: Current trash removal is leveraging HDFS trash removal. This subtask can be used to evaluate performance impact. specifically, evaluate * key removal in presence of a large number of buckets vs few buckets * key removal in presence of a large number of keys vs few keys was: Current trash removal is leveraging HDFS trash removal. This subtrask can be used to evaluate performance impact. specifically evaluate * key removal in presence of large number of buckets vs few buckets * key removal in presence of large number of keys vs few keys > Performance evaluation for Ozone trash emptier > -- > > Key: HDDS-4348 > URL: https://issues.apache.org/jira/browse/HDDS-4348 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Prashant Pogde >Assignee: Sadanand Shenoy >Priority: Major > > Current trash removal is leveraging HDFS trash removal. This subtask can be > used to evaluate performance impact. > specifically, evaluate > * key removal in presence of a large number of buckets vs few buckets > * key removal in presence of a large number of keys vs few keys -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4348) Performance evaluation for Ozone trash emptier
[ https://issues.apache.org/jira/browse/HDDS-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian updated HDDS-4348: - Summary: Performance evaluation for Ozone trash emptier (was: Perf evaluation fopr Ozone trash emptier) > Performance evaluation for Ozone trash emptier > -- > > Key: HDDS-4348 > URL: https://issues.apache.org/jira/browse/HDDS-4348 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Prashant Pogde >Assignee: Sadanand Shenoy >Priority: Major > > Current trash removal is leveraging HDFS trash removal. This subtrask can be > used to evaluate performance impact. > specifically evaluate > * key removal in presence of large number of buckets vs few buckets > * key removal in presence of large number of keys vs few keys -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4328) Provide fallback cache restore key
[
https://issues.apache.org/jira/browse/HDDS-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-4328:
---
Status: Patch Available (was: In Progress)
> Provide fallback cache restore key
> --
>
> Key: HDDS-4328
> URL: https://issues.apache.org/jira/browse/HDDS-4328
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: build
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
>
> Maven dependency cache hit or miss in GitHub Actions workflow is based on the
> hash of all POM files. If any POM is changed, all dependencies need to be
> downloaded from scratch. Providing {{restore-keys}} would allow it to fall
> back to one of the previous caches, potentially avoiding most of the
> downloads.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on pull request #1503: URL: https://github.com/apache/hadoop-ozone/pull/1503#issuecomment-712282977 > @rakeshadr , thanks for working on this. Initial review comments from me. > Please have a look for this when you get a chance. Thanks a lot for the review comments. I will update the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1503: HDDS-4332: ListFileStatus - do lookup in directory and file tables
rakeshadr commented on a change in pull request #1503:
URL: https://github.com/apache/hadoop-ozone/pull/1503#discussion_r50720
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##
@@ -583,4 +588,112 @@ public static OmKeyInfo getOmKeyInfoFromFileTable(boolean
openFileTable,
return dbOmKeyInfo;
}
+ /**
+ * Gets OmKeyInfo if exists for the given key name in the DB.
+ *
+ * @param omMetadataMgr metadata manager
+ * @param volumeNamevolume name
+ * @param bucketNamebucket name
+ * @param keyName key name
+ * @param scmBlockSize scm block size
+ * @return OzoneFileStatus
+ * @throws IOException DB failure
+ */
+ @Nullable
+ public static OzoneFileStatus getOMKeyInfoIfExists(
+ OMMetadataManager omMetadataMgr, String volumeName, String bucketName,
+ String keyName, long scmBlockSize) throws IOException {
+
+Path keyPath = Paths.get(keyName);
+Iterator elements = keyPath.iterator();
+String bucketKey = omMetadataMgr.getBucketKey(volumeName, bucketName);
+OmBucketInfo omBucketInfo =
+omMetadataMgr.getBucketTable().get(bucketKey);
+
+long lastKnownParentId = omBucketInfo.getObjectID();
+OmDirectoryInfo omDirInfo = null;
+while (elements.hasNext()) {
+ String fileName = elements.next().toString();
+
+ // For example, /vol1/buck1/a/b/c/d/e/file1.txt
+ // 1. Do lookup path component on directoryTable starting from bucket
+ // 'buck1' to the leaf node component, which is 'file1.txt'.
+ // 2. If there is no dir exists for the leaf node component 'file1.txt'
+ // then do look it on fileTable.
+ String dbNodeName = omMetadataMgr.getOzonePathKey(
+ lastKnownParentId, fileName);
+ omDirInfo = omMetadataMgr.getDirectoryTable().get(dbNodeName);
+
+ if (omDirInfo != null) {
+lastKnownParentId = omDirInfo.getObjectID();
+ } else if (!elements.hasNext()) {
+// reached last path component. Check file exists for the given path.
+OmKeyInfo omKeyInfo = OMFileRequest.getOmKeyInfoFromFileTable(false,
+omMetadataMgr, dbNodeName, keyName);
+if (omKeyInfo != null) {
+ return new OzoneFileStatus(omKeyInfo, scmBlockSize, false);
+}
+ } else {
+// Missing intermediate directory and just return null;
+// key not found in DB
+return null;
+ }
+}
+
+if (omDirInfo != null) {
+ OmKeyInfo omKeyInfo = getOmKeyInfo(volumeName, bucketName, omDirInfo,
+ keyName);
+ return new OzoneFileStatus(omKeyInfo, scmBlockSize, true);
+}
+
+// key not found in DB
+return null;
+ }
+
+ /**
+ * Prepare OmKeyInfo from OmDirectoryInfo.
+ *
+ * @param volumeName volume name
+ * @param bucketName bucket name
+ * @param dirInfodirectory info
+ * @param keyNameusewr given key name
+ * @return OmKeyInfo object
+ */
+ @NotNull
+ public static OmKeyInfo getOmKeyInfo(String volumeName, String bucketName,
+ OmDirectoryInfo dirInfo, String keyName) {
+
+OmKeyInfo.Builder builder = new OmKeyInfo.Builder();
+builder.setParentObjectID(dirInfo.getParentObjectID());
+builder.setKeyName(keyName);
+builder.setAcls(dirInfo.getAcls());
+builder.addAllMetadata(dirInfo.getMetadata());
+builder.setVolumeName(volumeName);
+builder.setBucketName(bucketName);
+builder.setCreationTime(dirInfo.getCreationTime());
+builder.setModificationTime(dirInfo.getModificationTime());
+builder.setObjectID(dirInfo.getObjectID());
+builder.setUpdateID(dirInfo.getUpdateID());
+builder.setFileName(dirInfo.getName());
+builder.setReplicationType(HddsProtos.ReplicationType.RATIS);
+builder.setReplicationFactor(HddsProtos.ReplicationFactor.ONE);
Review comment:
I've followed existing logic during directory creation :
[OMDirectoryCreateRequest.java#L351](https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java#L351).
OzoneConfigKeys.OZONE_REPLICATION_TYPE_DEFAULT is string object but
OmKeyInfo.Builder().setReplicationType(HddsProtos.ReplicationType.RATIS)
requires `enum ReplicationType`. Changing code requires a conversion logic.
Also, OZONE_REPLICATION_DEFAULT is THREE but directory has a value ONE.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoo
[jira] [Resolved] (HDDS-4324) DatanodeAdminMonitor no longers needs maintenance end time to be passed
[
https://issues.apache.org/jira/browse/HDDS-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell resolved HDDS-4324.
-
Fix Version/s: 1.1.0
Resolution: Fixed
> DatanodeAdminMonitor no longers needs maintenance end time to be passed
> ---
>
> Key: HDDS-4324
> URL: https://issues.apache.org/jira/browse/HDDS-4324
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: SCM
>Affects Versions: 1.1.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> An earlier change moved the maintenance endtime into the NodeStatus object.
> However when adding a node to the decommission monitor the end time must
> still be passed. This value is never used.
> This Jira will remove the endInHours field from the interface:
> {code}
> public interface DatanodeAdminMonitor extends Runnable {
> void startMonitoring(DatanodeDetails dn, int endInHours);
> void stopMonitoring(DatanodeDetails dn);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel merged pull request #1500: HDDS-4324. DatanodeAdminMonitor no longers needs maintenance end time to be passed
sodonnel merged pull request #1500: URL: https://github.com/apache/hadoop-ozone/pull/1500 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4307) Start Trash Emptier in Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213327#comment-17213327 ] YiSheng Lien edited comment on HDDS-4307 at 10/19/20, 4:16 PM: --- Thank you for the reply. I would upload it ASAP. Update: Hi [~sadanand_shenoy], could you be so kind as to give me your slack name or email to me? (The design-doc is near complete, I would invite you to the doc if you don't mind, thanks) Update II: Here is the [doc|https://docs.google.com/document/d/1fpmmnuhcCI-ATZfhhQmADq1J49Kawi9_UffcLVeuabQ/edit?usp=sharing]. Feel free to review it and give me some feedback, thanks was (Author: cxorm): Thank you for the reply. I would upload it ASAP. Update: Hi [~sadanand_shenoy], could you be so kind as to give me your slack name or email to me? (The design-doc is near complete, I would invite you to the doc if you don't mind, thanks) Update II: Here is the [doc|https://docs.google.com/document/d/1fpmmnuhcCI-ATZfhhQmADq1J49Kawi9_UffcLVeuabQ/edit?usp=sharing]. > Start Trash Emptier in Ozone Manager > > > Key: HDDS-4307 > URL: https://issues.apache.org/jira/browse/HDDS-4307 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4307) Start Trash Emptier in Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213327#comment-17213327 ] YiSheng Lien edited comment on HDDS-4307 at 10/19/20, 4:14 PM: --- Thank you for the reply. I would upload it ASAP. Update: Hi [~sadanand_shenoy], could you be so kind as to give me your slack name or email to me? (The design-doc is near complete, I would invite you to the doc if you don't mind, thanks) Update II: Here is the [doc|https://docs.google.com/document/d/1fpmmnuhcCI-ATZfhhQmADq1J49Kawi9_UffcLVeuabQ/edit?usp=sharing]. was (Author: cxorm): Thank you for the reply. I would upload it ASAP. Update: Hi [~sadanand_shenoy], could you be so kind as to give me your slack name or email to me? (The design-doc is near complete, I would invite you to the doc if you don't mind, thanks) > Start Trash Emptier in Ozone Manager > > > Key: HDDS-4307 > URL: https://issues.apache.org/jira/browse/HDDS-4307 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cxorm edited a comment on pull request #1301: HDDS-3882. Update modification time when updating volume/bucket/key ACLs
cxorm edited a comment on pull request #1301: URL: https://github.com/apache/hadoop-ozone/pull/1301#issuecomment-711871349 > Thanks @cxorm for the update. The latest PR LGTM overall, a few comments added inline. Thank you @xiaoyuyao for the careful review. The fix was added to address the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-2413) Set configuration variables from annotated java objects
[
https://issues.apache.org/jira/browse/HDDS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai reassigned HDDS-2413:
--
Assignee: Attila Doroszlai (was: Marton Elek)
> Set configuration variables from annotated java objects
> ---
>
> Key: HDDS-2413
> URL: https://issues.apache.org/jira/browse/HDDS-2413
> Project: Hadoop Distributed Data Store
> Issue Type: Task
>Reporter: Marton Elek
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.0.0
>
>
> HDDS-1469 introduced a new method to handle configuration. Configuration can
> be injected directly to java objects which makes all the java constants
> unnecessary.
>
> Almost.
>
> To read the configuration it's enough to have an annotated java object. For
> example:
>
> {code:java}
> @ConfigGroup(prefix = "hdds.scm")
> public class ScmConfig {
> private String principal;
> private String keytab; @Config(key = "kerberos.principal",
> type = ConfigType.STRING,
> defaultValue = "",
> tags = { ConfigTag.SECURITY, ConfigTag.OZONE },
> description = "This Kerberos principal is used by the SCM service."
> )
> public void setKerberosPrincipal(String kerberosPrincipal) {
> this.principal = kerberosPrincipal; {code}
> And the configuration can be set in ozone-site.xml
> Unfortunately during the unit testing we need to inject the configuration
> variables programmatically which requires a String constant:
> {code:java}
> configuration.set(ScmConfig.ConfigStrings.HDDS_SCM_KERBEROS_PRINCIPAL_KEY,
>
> "scm/" + host + "@" + realm); {code}
> I propose to implement a simple setter in the OzoneConfiguration which may
> help to set configuration based on an annotated configuration object instance:
> {code:java}
> OzoneConfiguration conf = new OzoneConfiguration();
> SCMHTTPServerConfig httpConfig = SCMHTTPServerConfig(principal1,...);
> conf.setFromObject(httpConfig){code}
> This is the opposite direction of the existing OzoneConfiguration.getObject()
> and can be implemented with a similar approach.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4158) Provide a class type for Java based configuration
[ https://issues.apache.org/jira/browse/HDDS-4158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai resolved HDDS-4158. Fix Version/s: 1.1.0 Assignee: maobaolong Resolution: Implemented > Provide a class type for Java based configuration > - > > Key: HDDS-4158 > URL: https://issues.apache.org/jira/browse/HDDS-4158 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4158) Provide a class type for Java based configuration
[ https://issues.apache.org/jira/browse/HDDS-4158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-4158: --- Labels: (was: pull-request-available) > Provide a class type for Java based configuration > - > > Key: HDDS-4158 > URL: https://issues.apache.org/jira/browse/HDDS-4158 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 1.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4158) Provide a class type for Java based configuration
[ https://issues.apache.org/jira/browse/HDDS-4158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-4158: --- Description: Provide a class type for Java based configuration. > Provide a class type for Java based configuration > - > > Key: HDDS-4158 > URL: https://issues.apache.org/jira/browse/HDDS-4158 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 1.1.0 > > > Provide a class type for Java based configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1407: HDDS-4158. Provide a class type for Java based configuration
adoroszlai commented on pull request #1407: URL: https://github.com/apache/hadoop-ozone/pull/1407#issuecomment-712249238 Thanks @maobaolong for the improvement and @elek for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai merged pull request #1407: HDDS-4158. Provide a class type for Java based configuration
adoroszlai merged pull request #1407: URL: https://github.com/apache/hadoop-ozone/pull/1407 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4328) Provide fallback cache restore key
[
https://issues.apache.org/jira/browse/HDDS-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-4328:
-
Labels: pull-request-available (was: )
> Provide fallback cache restore key
> --
>
> Key: HDDS-4328
> URL: https://issues.apache.org/jira/browse/HDDS-4328
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: build
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
>
> Maven dependency cache hit or miss in GitHub Actions workflow is based on the
> hash of all POM files. If any POM is changed, all dependencies need to be
> downloaded from scratch. Providing {{restore-keys}} would allow it to fall
> back to one of the previous caches, potentially avoiding most of the
> downloads.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #1505: HDDS-4328. Provide fallback cache restore key
adoroszlai opened a new pull request #1505: URL: https://github.com/apache/hadoop-ozone/pull/1505 ## What changes were proposed in this pull request? Specify fallback cache restore key (`restore-keys`) for GitHub Actions CI workflow. Maven dependency cache hit or miss in GitHub Actions workflow is based on the hash of all POM files. If any POM is changed, all dependencies need to be downloaded from scratch. The fallback key allows it to use one of the previous caches, potentially avoiding most of the downloads ([doc](https://docs.github.com/en/free-pro-team@latest/actions/guides/caching-dependencies-to-speed-up-workflows#matching-a-cache-key)). https://issues.apache.org/jira/browse/HDDS-4328 ## How was this patch tested? Regular CI in two steps. Initial [commit](https://github.com/adoroszlai/hadoop-ozone/commit/582d89af0009e909fbcbaf82e359d0068924de79): ``` key: maven-repo-91e13458cfebf5c61e6f829853e6d98af55898ed81a5ad0b4e855c96fc0fe0ce restore-keys: maven-repo- Cache not found for input keys: maven-repo-91e13458cfebf5c61e6f829853e6d98af55898ed81a5ad0b4e855c96fc0fe0ce, maven-repo- ... Post job cleanup. Cache saved successfully ``` Next [commit](https://github.com/adoroszlai/hadoop-ozone/commit/45730362fab01d9a0ed4bcf62b1f6ac8dec5c965), with POM changes: ``` key: maven-repo-5fb5c783db98f682a1b3f4926097ed49d2297d6ee0118ace1d994e545b0259bd restore-keys: maven-repo- ... Cache restored from key: maven-repo-91e13458cfebf5c61e6f829853e6d98af55898ed81a5ad0b4e855c96fc0fe0ce ``` The second commit encountered cache miss, but restored from previous version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4297) Allow multiple transactions per container to be sent for deletion by SCM
[ https://issues.apache.org/jira/browse/HDDS-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain resolved HDDS-4297. --- Fix Version/s: 1.1.0 Resolution: Fixed > Allow multiple transactions per container to be sent for deletion by SCM > > > Key: HDDS-4297 > URL: https://issues.apache.org/jira/browse/HDDS-4297 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > Currently SCM Block Deleting Service allows only one transaction per > container to be sent for deletion to the datanode. This can slow down > deletion if there are multiple delete transactions for a container. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 closed pull request #1459: HDDS-4297. Allow multiple transactions per container to be sent for deletion by SCM.
lokeshj1703 closed pull request #1459: URL: https://github.com/apache/hadoop-ozone/pull/1459 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on pull request #1459: HDDS-4297. Allow multiple transactions per container to be sent for deletion by SCM.
lokeshj1703 commented on pull request #1459: URL: https://github.com/apache/hadoop-ozone/pull/1459#issuecomment-712178445 @bshashikant @linyiqun Thanks for reviewing the PR! I have committed it to master branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] linyiqun commented on pull request #1459: HDDS-4297. Allow multiple transactions per container to be sent for deletion by SCM.
linyiqun commented on pull request #1459: URL: https://github.com/apache/hadoop-ozone/pull/1459#issuecomment-712021874 Good to know the testing result. +1 for this change. Thanks @lokeshj1703 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4079) Create ZH translation of Containers.md in doc
[ https://issues.apache.org/jira/browse/HDDS-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yu reassigned HDDS-4079: - Assignee: Yang Yu > Create ZH translation of Containers.md in doc > - > > Key: HDDS-4079 > URL: https://issues.apache.org/jira/browse/HDDS-4079 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Marton Elek >Assignee: Yang Yu >Priority: Major > Labels: translation-zh > > HDDS-4042 modified the documentation page. zh translation should be updated > for changed / new files. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4355) Deleted container is marked as missing on recon UI
[
https://issues.apache.org/jira/browse/HDDS-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HDDS-4355:
-
Description:
{noformat}
~/ozoneenv/ozone]$ bin/ozone admin container info 104825
Container id: 104825
Pipeline id: 10955a24-2047-416f-85ac-94523cfe8d40
Container State: DELETED
Datanodes: []
{noformat}
was:
{noformat}
~/ozoneenv/ozone]$ bin/ozone admin container info 104825
Container id: 104825
Pipeline id: 10955a24-2047-416f-85ac-94523cfe8d40
Container State: DELETED
Datanodes: []
{noformat}
!screenshot-1.png!
> Deleted container is marked as missing on recon UI
> --
>
> Key: HDDS-4355
> URL: https://issues.apache.org/jira/browse/HDDS-4355
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Priority: Major
> Attachments: screenshot-1.png
>
>
> {noformat}
> ~/ozoneenv/ozone]$ bin/ozone admin container info 104825
> Container id: 104825
> Pipeline id: 10955a24-2047-416f-85ac-94523cfe8d40
> Container State: DELETED
> Datanodes: []
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4355) Deleted container is marked as missing on recon UI
[
https://issues.apache.org/jira/browse/HDDS-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HDDS-4355:
-
Attachment: screenshot-1.png
> Deleted container is marked as missing on recon UI
> --
>
> Key: HDDS-4355
> URL: https://issues.apache.org/jira/browse/HDDS-4355
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Priority: Major
> Attachments: screenshot-1.png
>
>
> {noformat}
> ~/ozoneenv/ozone]$ bin/ozone admin container info 104825
> Container id: 104825
> Pipeline id: 10955a24-2047-416f-85ac-94523cfe8d40
> Container State: DELETED
> Datanodes: []
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4355) Deleted container is marked as missing on recon UI
[
https://issues.apache.org/jira/browse/HDDS-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HDDS-4355:
-
Description:
{noformat}
~/ozoneenv/ozone]$ bin/ozone admin container info 104825
Container id: 104825
Pipeline id: 10955a24-2047-416f-85ac-94523cfe8d40
Container State: DELETED
Datanodes: []
{noformat}
!screenshot-1.png!
was:
{noformat}
~/ozoneenv/ozone]$ bin/ozone admin container info 104825
Container id: 104825
Pipeline id: 10955a24-2047-416f-85ac-94523cfe8d40
Container State: DELETED
Datanodes: []
{noformat}
> Deleted container is marked as missing on recon UI
> --
>
> Key: HDDS-4355
> URL: https://issues.apache.org/jira/browse/HDDS-4355
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Priority: Major
> Attachments: screenshot-1.png
>
>
> {noformat}
> ~/ozoneenv/ozone]$ bin/ozone admin container info 104825
> Container id: 104825
> Pipeline id: 10955a24-2047-416f-85ac-94523cfe8d40
> Container State: DELETED
> Datanodes: []
> {noformat}
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4355) Deleted container is marked as missing on recon UI
[
https://issues.apache.org/jira/browse/HDDS-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated HDDS-4355:
-
Description:
{noformat}
~/ozoneenv/ozone]$ bin/ozone admin container info 104825
Container id: 104825
Pipeline id: 10955a24-2047-416f-85ac-94523cfe8d40
Container State: DELETED
Datanodes: []
{noformat}
was:
[ozoneadmin@tdw-10-51-87-181 ~/ozoneenv/ozone]$ bin/ozone admin container info
385122
Container id: 385122
Pipeline id: e7823c96-3562-43d3-a058-4d259ac6d54e
Container State: DELETED
Datanodes: []
> Deleted container is marked as missing on recon UI
> --
>
> Key: HDDS-4355
> URL: https://issues.apache.org/jira/browse/HDDS-4355
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Priority: Major
>
> {noformat}
> ~/ozoneenv/ozone]$ bin/ozone admin container info 104825
> Container id: 104825
> Pipeline id: 10955a24-2047-416f-85ac-94523cfe8d40
> Container State: DELETED
> Datanodes: []
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4355) Deleted container is marked as missing on recon UI
Sammi Chen created HDDS-4355: Summary: Deleted container is marked as missing on recon UI Key: HDDS-4355 URL: https://issues.apache.org/jira/browse/HDDS-4355 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Sammi Chen [ozoneadmin@tdw-10-51-87-181 ~/ozoneenv/ozone]$ bin/ozone admin container info 385122 Container id: 385122 Pipeline id: e7823c96-3562-43d3-a058-4d259ac6d54e Container State: DELETED Datanodes: [] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on a change in pull request #1502: HDDS-4354. Avoid spamming of logs because of deleted transactions.
lokeshj1703 commented on a change in pull request #1502:
URL: https://github.com/apache/hadoop-ozone/pull/1502#discussion_r507591562
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImpl.java
##
@@ -130,9 +130,7 @@ public void incrementCount(List txIDs) throws
IOException {
DeletedBlocksTransaction block =
scmMetadataStore.getDeletedBlocksTXTable().get(txID);
if (block == null) {
- // Should we make this an error ? How can we not find the deleted
- // TXID?
- LOG.warn("Deleted TXID {} not found.", txID);
+ LOG.debug("Deleted TXID {} not found.", txID);
Review comment:
NIT: We can include the log inside check for debug log enabled.
```suggestion
if (LOG.isDebugEnabled()) {
LOG.debug("Deleted TXID {} not found.", txID);
}
```
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/lease/LeaseManager.java
##
@@ -235,7 +235,7 @@ public void run() {
}
} catch (InterruptedException e) {
// This means a new lease is added to activeLeases.
- LOG.error("Execution was interrupted ", e);
+ LOG.debug("Execution was interrupted ", e);
Review comment:
This should not be related to block deletion. What exception are we
seeing here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cxorm commented on pull request #1301: HDDS-3882. Update modification time when updating volume/bucket/key ACLs
cxorm commented on pull request #1301: URL: https://github.com/apache/hadoop-ozone/pull/1301#issuecomment-711871349 > Thanks @cxorm for the update. The latest PR LGTM overall, a few comments added inline. Thank you @xiaoyuyao for the careful review. The fix was added to addressed the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on pull request #1459: HDDS-4297. Allow multiple transactions per container to be sent for deletion by SCM.
lokeshj1703 commented on pull request #1459: URL: https://github.com/apache/hadoop-ozone/pull/1459#issuecomment-711778460 @linyiqun currently we allow block deletion for only closed containers. For open containers deletion needs to be done via ratis pipeline which is not currently implemented. So there is no block allocation advantage right now. I do not think there is a huge window where containers are open and are simultaneously receiving block deletion requests in bulk. Further it is better to accumulate transactions of a container if possible. There is an overhead to handling a block deletion request for datanode. It needs to load rockdb instance and then add those transactions to the db. It is better for datanode to process multiple such transactions at once. I tested the PR and found that it gives better deletion speed for deleting a milllion keys. Without PR - SCM sends total 180 blocks for deletion in a minute. This number does not increase even after increasing configs. Datanode deletion speed was very slow. I monitored a datanode disk space and it deleted less than a GB even after 5 minutes. With PR - SCM sends 6000 blocks for deletion. Datanode was deleting 1 GB of data every minute. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3188) Add failover proxy to SCM block protocol
[ https://issues.apache.org/jira/browse/HDDS-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216508#comment-17216508 ] Li Cheng commented on HDDS-3188: PR is merged. Resolving > Add failover proxy to SCM block protocol > > > Key: HDDS-3188 > URL: https://issues.apache.org/jira/browse/HDDS-3188 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Major > Labels: pull-request-available > > Need to supports 2N + 1 SCMs. Add configs and logic to support multiple SCMs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3188) Add failover proxy to SCM block protocol
[ https://issues.apache.org/jira/browse/HDDS-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng updated HDDS-3188: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Add failover proxy to SCM block protocol > > > Key: HDDS-3188 > URL: https://issues.apache.org/jira/browse/HDDS-3188 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Major > Labels: pull-request-available > > Need to supports 2N + 1 SCMs. Add configs and logic to support multiple SCMs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4192) enable SCM Raft Group based on config ozone.scm.names
[ https://issues.apache.org/jira/browse/HDDS-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216507#comment-17216507 ] Li Cheng commented on HDDS-4192: PR is merged. Thanks [~glengeng] for the contribution. Resolving > enable SCM Raft Group based on config ozone.scm.names > - > > Key: HDDS-4192 > URL: https://issues.apache.org/jira/browse/HDDS-4192 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Glen Geng >Assignee: Glen Geng >Priority: Major > Labels: pull-request-available > > > Say ozone.scm.names is "ip1,ip2,ip3", scm with ip1 identifies its RaftPeerId > as scm1, scm with ip2 identifies its RaftPeerId as scm2, scm with ip3 > identifies its RaftPeerId as scm3. They will automatically become a raft > group. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4192) enable SCM Raft Group based on config ozone.scm.names
[ https://issues.apache.org/jira/browse/HDDS-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng resolved HDDS-4192. Resolution: Fixed > enable SCM Raft Group based on config ozone.scm.names > - > > Key: HDDS-4192 > URL: https://issues.apache.org/jira/browse/HDDS-4192 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Glen Geng >Assignee: Glen Geng >Priority: Major > Labels: pull-request-available > > > Say ozone.scm.names is "ip1,ip2,ip3", scm with ip1 identifies its RaftPeerId > as scm1, scm with ip2 identifies its RaftPeerId as scm2, scm with ip3 > identifies its RaftPeerId as scm3. They will automatically become a raft > group. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
