[GitHub] [hadoop-ozone] codecov-commenter commented on pull request #1331: HDDS-4117. Normalize Keypath for listKeys.
codecov-commenter commented on pull request #1331: URL: https://github.com/apache/hadoop-ozone/pull/1331#issuecomment-673757256 # [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1331?src=pr=h1) Report > Merging [#1331](https://codecov.io/gh/apache/hadoop-ozone/pull/1331?src=pr=desc) into [master](https://codecov.io/gh/apache/hadoop-ozone/commit/9c2218047bf8cf13f9c52ad819c938d9ba964b81=desc) will **decrease** coverage by `0.06%`. > The diff coverage is `100.00%`. [](https://codecov.io/gh/apache/hadoop-ozone/pull/1331?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#1331 +/- ## - Coverage 74.41% 74.35% -0.07% - Complexity1030110324 +23 Files 984 985 +1 Lines 5039450549 +155 Branches 4914 4937 +23 + Hits 3750337587 +84 - Misses1053110600 +69 - Partials 2360 2362 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/hadoop-ozone/pull/1331?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...src/main/java/org/apache/hadoop/ozone/OmUtils.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1331/diff?src=pr=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL09tVXRpbHMuamF2YQ==) | `80.66% <100.00%> (+1.08%)` | `49.00 <3.00> (+3.00)` | | | [...ava/org/apache/hadoop/ozone/om/KeyManagerImpl.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1331/diff?src=pr=tree#diff-aGFkb29wLW96b25lL296b25lLW1hbmFnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9vbS9LZXlNYW5hZ2VySW1wbC5qYXZh) | `65.34% <100.00%> (+0.43%)` | `140.00 <4.00> (+4.00)` | | | [...pache/hadoop/ozone/om/request/OMClientRequest.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1331/diff?src=pr=tree#diff-aGFkb29wLW96b25lL296b25lLW1hbmFnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9vbS9yZXF1ZXN0L09NQ2xpZW50UmVxdWVzdC5qYXZh) | `81.17% <100.00%> (-1.25%)` | `29.00 <1.00> (-2.00)` | | | [...hdds/scm/container/common/helpers/ExcludeList.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1331/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy9zY20vY29udGFpbmVyL2NvbW1vbi9oZWxwZXJzL0V4Y2x1ZGVMaXN0LmphdmE=) | `78.26% <0.00%> (-21.74%)` | `17.00% <0.00%> (-5.00%)` | | | [...doop/ozone/om/exceptions/OMNotLeaderException.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1331/diff?src=pr=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL29tL2V4Y2VwdGlvbnMvT01Ob3RMZWFkZXJFeGNlcHRpb24uamF2YQ==) | `48.48% <0.00%> (-18.19%)` | `3.00% <0.00%> (-2.00%)` | | | [.../apache/hadoop/hdds/scm/node/StaleNodeHandler.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1331/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL25vZGUvU3RhbGVOb2RlSGFuZGxlci5qYXZh) | `88.88% <0.00%> (-11.12%)` | `4.00% <0.00%> (ø%)` | | | [...e/hadoop/ozone/recon/tasks/OMDBUpdatesHandler.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1331/diff?src=pr=tree#diff-aGFkb29wLW96b25lL3JlY29uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3Avb3pvbmUvcmVjb24vdGFza3MvT01EQlVwZGF0ZXNIYW5kbGVyLmphdmE=) | `60.93% <0.00%> (-9.66%)` | `9.00% <0.00%> (-8.00%)` | | | [...ache/hadoop/ozone/om/codec/S3SecretValueCodec.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1331/diff?src=pr=tree#diff-aGFkb29wLW96b25lL296b25lLW1hbmFnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9vbS9jb2RlYy9TM1NlY3JldFZhbHVlQ29kZWMuamF2YQ==) | `90.90% <0.00%> (-9.10%)` | `3.00% <0.00%> (-1.00%)` | | | [...va/org/apache/hadoop/hdds/utils/db/RDBMetrics.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1331/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvZnJhbWV3b3JrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy91dGlscy9kYi9SREJNZXRyaWNzLmphdmE=) | `92.85% <0.00%> (-7.15%)` | `14.00% <0.00%> (-1.00%)` | | | [...ozone/container/ozoneimpl/ContainerController.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1331/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvb3pvbmVpbXBsL0NvbnRhaW5lckNvbnRyb2xsZXIuamF2YQ==) | `76.31% <0.00%> (-5.27%)` | `13.00% <0.00%> (-1.00%)` | | | ... and [44 more](https://codecov.io/gh/apache/hadoop-ozone/pull/1331/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1331?src=pr=continue). > **Legend** - [Click here to
[jira] [Updated] (HDDS-4117) Normalize Keypath for listKeys.
[ https://issues.apache.org/jira/browse/HDDS-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-4117: - Status: Patch Available (was: Open) > Normalize Keypath for listKeys. > > > Key: HDDS-4117 > URL: https://issues.apache.org/jira/browse/HDDS-4117 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > When ozone.om.enable.filesystem.paths, OM normalizes path, and stores the > Keyname. > When listKeys uses given keyName(not normalized key path) as prefix and > Starkey the list-keys will return empty result. > Similar to HDDS-4102, we should normalize startKey and keyPrefix. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4117) Normalize Keypath for listKeys.
[ https://issues.apache.org/jira/browse/HDDS-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4117: - Labels: pull-request-available (was: ) > Normalize Keypath for listKeys. > > > Key: HDDS-4117 > URL: https://issues.apache.org/jira/browse/HDDS-4117 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > When ozone.om.enable.filesystem.paths, OM normalizes path, and stores the > Keyname. > When listKeys uses given keyName(not normalized key path) as prefix and > Starkey the list-keys will return empty result. > Similar to HDDS-4102, we should normalize startKey and keyPrefix. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #1331: HDDS-4117.
bharatviswa504 opened a new pull request #1331: URL: https://github.com/apache/hadoop-ozone/pull/1331 ## What changes were proposed in this pull request? Normalize Keypath for listKeys. When ozone.om.enable.filesystem.paths, OM normalizes path, and stores the Keyname. When listKeys uses given keyName(not normalized key path) as prefix and Starkey the list-keys will return empty result. Similar to HDDS-4102, we should normalize startKey and keyPrefix. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4117 ## How was this patch tested? Added a test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4117) Normalize Keypath for listKeys.
[ https://issues.apache.org/jira/browse/HDDS-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-4117: - Labels: (was: pull-request-available) > Normalize Keypath for listKeys. > > > Key: HDDS-4117 > URL: https://issues.apache.org/jira/browse/HDDS-4117 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > When ozone.om.enable.filesystem.paths, OM normalizes path, and stores the > Keyname. > When listKeys uses given keyName(not normalized key path) as prefix and > Starkey the list-keys will return empty result. > Similar to HDDS-4102, we should normalize startKey and keyPrefix. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4117) Normalize Keypath for listKeys.
[ https://issues.apache.org/jira/browse/HDDS-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-4117: - Description: When ozone.om.enable.filesystem.paths, OM normalizes path, and stores the Keyname. When listKeys uses given keyName(not normalized key path) as prefix and Starkey the list-keys will return empty result. Similar to HDDS-4102, we should normalize startKey and keyPrefix. was: When ozone.om.enable.filesystem.paths, OM normalizes path, and stores the Keyname. Now when user tries to read the file from S3 using the keyName which user has used to create the Key, it will return error KEY_NOT_FOUND The issue is, lookupKey need to normalize path, when ozone.om.enable.filesystem.paths is enabled. This is common API used by S3/FS. > Normalize Keypath for listKeys. > > > Key: HDDS-4117 > URL: https://issues.apache.org/jira/browse/HDDS-4117 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > When ozone.om.enable.filesystem.paths, OM normalizes path, and stores the > Keyname. > When listKeys uses given keyName(not normalized key path) as prefix and > Starkey the list-keys will return empty result. > Similar to HDDS-4102, we should normalize startKey and keyPrefix. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4117) Normalize Keypath for listKeys.
Bharat Viswanadham created HDDS-4117: Summary: Normalize Keypath for listKeys. Key: HDDS-4117 URL: https://issues.apache.org/jira/browse/HDDS-4117 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham When ozone.om.enable.filesystem.paths, OM normalizes path, and stores the Keyname. Now when user tries to read the file from S3 using the keyName which user has used to create the Key, it will return error KEY_NOT_FOUND The issue is, lookupKey need to normalize path, when ozone.om.enable.filesystem.paths is enabled. This is common API used by S3/FS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on pull request #1327: HDDS-4098. Improve om admin getserviceroles error message
dineshchitlangia commented on pull request #1327: URL: https://github.com/apache/hadoop-ozone/pull/1327#issuecomment-673723932 @dshavkanihort Thank you for working on this. Overall the changes look good to me, I see a checkstyle issue has been reported that needs to be fixed. @adoroszlai do you have any suggestion for improving the wording of the message? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4074) [OFS] Implement AbstractFileSystem for RootedOzoneFileSystem
[
https://issues.apache.org/jira/browse/HDDS-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-4074:
-
Labels: pull-request-available (was: )
> [OFS] Implement AbstractFileSystem for RootedOzoneFileSystem
>
>
> Key: HDDS-4074
> URL: https://issues.apache.org/jira/browse/HDDS-4074
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Filesystem
>Reporter: Attila Doroszlai
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
>
> Extracted from HDDS-3805: introduce an implementation of
> {{AbstractFileSystem}}, similar to {{OzFs}}, for {{RootedOzoneFileSystem}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #1330: HDDS-4074. [OFS] Implement AbstractFileSystem for RootedOzoneFileSystem
adoroszlai opened a new pull request #1330:
URL: https://github.com/apache/hadoop-ozone/pull/1330
## What changes were proposed in this pull request?
Add `RootedOzFs`, implementation of `AbstractFileSystem` for
`RootedOzoneFileSystem`. This will allow running MapReduce tests with OFS.
Extracted from #1088. Also applied the fix from HDDS-3482.
https://issues.apache.org/jira/browse/HDDS-4074
## How was this patch tested?
Executed `ozone-mr` acceptance tests locally:
```
export OZONE_TEST_SELECTOR='ozone-mr\|hadoop..'
hadoop-ozone/dev-support/checks/acceptance.sh
```
after tweaking config and MR test to use `ofs` instead of `o3fs`:
```diff
diff --git hadoop-ozone/dist/src/main/compose/ozone-mr/common-config
hadoop-ozone/dist/src/main/compose/ozone-mr/common-config
index 46e752731..91588aa77 100644
--- hadoop-ozone/dist/src/main/compose/ozone-mr/common-config
+++ hadoop-ozone/dist/src/main/compose/ozone-mr/common-config
@@ -30,7 +30,8 @@ OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
HDFS-SITE.XML_dfs.datanode.address=0.0.0.0:1019
HDFS-SITE.XML_dfs.datanode.http.address=0.0.0.0:1012
-CORE-SITE.xml_fs.defaultFS=o3fs://bucket1.volume1/
+CORE-SITE.xml_fs.defaultFS=ofs://om/volume1/bucket1/
+CORE-SITE.XML_fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem
MAPRED-SITE.XML_mapreduce.framework.name=yarn
MAPRED-SITE.XML_yarn.app.mapreduce.am.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
diff --git hadoop-ozone/dist/src/main/smoketest/mapreduce.robot
hadoop-ozone/dist/src/main/smoketest/mapreduce.robot
index 654dd49c6..e2dfd11a1 100644
--- hadoop-ozone/dist/src/main/smoketest/mapreduce.robot
+++ hadoop-ozone/dist/src/main/smoketest/mapreduce.robot
@@ -39,5 +39,5 @@ Execute PI calculation
Execute WordCount
${exampleJar}Find example jar
${random}Generate Random String 2 [NUMBERS]
-${output} = Execute yarn jar
${exampleJar} wordcount o3fs://bucket1.volume1/key1
o3fs://bucket1.volume1/key1-${random}.count
+${output} = Execute yarn jar
${exampleJar} wordcount ofs://om/volume1/bucket1/key1
ofs://om/volume1/bucket1/key1-${random}.count
Should Contain ${output} completed
successfully
```
Regular CI:
https://github.com/adoroszlai/hadoop-ozone/runs/975699201
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] errose28 commented on a change in pull request #1298: HDDS-3869. Use different column families for datanode block and metadata
errose28 commented on a change in pull request #1298:
URL: https://github.com/apache/hadoop-ozone/pull/1298#discussion_r470210074
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/metadata/SchemaOneDeletedBlocksTable.java
##
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.container.metadata;
+
+import org.apache.hadoop.hdds.utils.MetadataKeyFilters;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.cache.CacheKey;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.hdds.utils.db.cache.CacheValue;
+import org.apache.hadoop.ozone.container.common.helpers.ChunkInfoList;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * For RocksDB instances written using DB schema version 1, all data is
+ * stored in the default column family. This differs from later schema
+ * versions, which put deleted blocks in a different column family.
+ * As a result, the block IDs used as keys for deleted blocks must be
+ * prefixed in schema version 1 so that they can be differentiated from
+ * regular blocks. However, these prefixes are not necessary in later schema
+ * versions, because the deleted blocks and regular blocks are in different
+ * column families.
+ *
+ * Since clients must operate independently of the underlying schema version,
+ * This class is returned to clients using {@link DatanodeStoreSchemaOneImpl}
+ * instances, allowing them to access keys as if no prefix is
+ * required, while it adds the prefix when necessary.
+ * This means the client should omit the deleted prefix when putting and
+ * getting keys, regardless of the schema version.
+ *
+ * Note that this class will only apply prefixes to keys as parameters,
+ * never as return types. This means that keys returned through iterators
+ * like {@link SchemaOneDeletedBlocksTable#getSequentialRangeKVs},
+ * {@link SchemaOneDeletedBlocksTable#getRangeKVs}, and
+ * {@link SchemaOneDeletedBlocksTable#iterator} will return keys prefixed
+ * with {@link SchemaOneDeletedBlocksTable#DELETED_KEY_PREFIX}.
+ */
+public class SchemaOneDeletedBlocksTable implements Table {
+ public static final String DELETED_KEY_PREFIX = "#deleted#";
+
+ private final Table table;
+
+ public SchemaOneDeletedBlocksTable(Table table) {
+this.table = table;
+ }
+
+ @Override
+ public void put(String key, ChunkInfoList value) throws IOException {
+table.put(prefix(key), value);
+ }
+
+ @Override
+ public void putWithBatch(BatchOperation batch, String key,
+ ChunkInfoList value)
+ throws IOException {
+table.putWithBatch(batch, prefix(key), value);
+ }
+
+ @Override
+ public boolean isEmpty() throws IOException {
+return table.isEmpty();
+ }
+
+ @Override
+ public void delete(String key) throws IOException {
+table.delete(prefix(key));
+ }
+
+ @Override
+ public void deleteWithBatch(BatchOperation batch, String key)
+ throws IOException {
+table.deleteWithBatch(batch, prefix(key));
+ }
+
+ /**
+ * Because the actual underlying table in this schema version is the
+ * default table where all keys are stored, this method will iterate
+ * through all keys in the database.
+ */
+ @Override
+ public TableIterator>
+ iterator() {
+return table.iterator();
+ }
Review comment:
In order to not support this method, the `KeyValueBlockIterator` will
need a different way to gain access to the iterator it uses internally when
filtering blocks by prefix. The updated version of the code will make this
class internal to the `AbstractDatanodeStore`, and callers can get it using
getters in the `DatanodeStore` interface that return the
`KeyValueBlockIterator`'s interface: `BlockIterator`. This way, the
`AbstractDatanodeStore` can initialize it with an iterator retrieved from the
block data table before wrapping it in a class to disable access to this method.
This is an automated message from the Apache
[jira] [Updated] (HDDS-4077) Incomplete OzoneFileSystem statistics
[ https://issues.apache.org/jira/browse/HDDS-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4077: - Labels: pull-request-available (was: ) > Incomplete OzoneFileSystem statistics > - > > Key: HDDS-4077 > URL: https://issues.apache.org/jira/browse/HDDS-4077 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: Attila Doroszlai >Assignee: Attila Doroszlai >Priority: Minor > Labels: pull-request-available > > OzoneFileSystem does not record some of the operations that are defined in > [Statistic|https://github.com/apache/hadoop-ozone/blob/d7ea4966656cfdb0b53a368eac52d71adb717104/hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/Statistic.java#L44-L75]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #1329: HDDS-4077. Incomplete OzoneFileSystem statistics
adoroszlai opened a new pull request #1329: URL: https://github.com/apache/hadoop-ozone/pull/1329 ## What changes were proposed in this pull request? Override some methods only for statistics. [S3AFileSystem](https://github.com/apache/hadoop/blob/a6df05bf5e24d04852a35b096c44e79f843f4776/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3003-L3054) also does so. https://issues.apache.org/jira/browse/HDDS-4077 ## How was this patch tested? Enabled DEBUG log for Ozone FS: ``` etc/hadoop/ozone-shell-log4j.properties 35:log4j.logger.org.apache.hadoop.fs.ozone=DEBUG etc/hadoop/log4j.properties 138:log4j.logger.org.apache.hadoop.fs.ozone=DEBUG ``` Ran `ozonefs.robot` and verified some previously missing operations are logged (for both OFS and O3FS). ``` 2020-08-12 12:42:38,057 [main] DEBUG ozone.OzoneFSStorageStatistics: op_glob_status += 1 - 1 ... 2020-08-12 12:42:51,163 [main] DEBUG ozone.OzoneFSStorageStatistics: op_mkdirs += 1 - 1 ... 2020-08-12 12:43:15,071 [main] DEBUG ozone.OzoneFSStorageStatistics: op_exists += 1 - 1 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1315: HDDS-4102. Normalize Keypath for lookupKey.
bharatviswa504 commented on pull request #1315: URL: https://github.com/apache/hadoop-ozone/pull/1315#issuecomment-673610964 Mistakenly for this PR branch was created in apache repo. Deleted my work in progress branch from apache repo, and created a new PR for this #1328 @elek I have replied to your comments in Jira. @rakeshadr Thanks for the review. Modified code to fix the test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1315: HDDS-4102. Normalize Keypath for lookupKey.
bharatviswa504 edited a comment on pull request #1315: URL: https://github.com/apache/hadoop-ozone/pull/1315#issuecomment-673610964 Mistakenly for this PR branch was created in apache repo. Deleted my work in progress branch from apache repo, and created a new PR for this #1328 @elek I have replied to your comments in Jira. @rakeshadr Thanks for the review. Modified code to fix the test. Please have a look into #1328 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #1328: HDDS-4102. Normalize Keypath for lookupKey.
bharatviswa504 opened a new pull request #1328: URL: https://github.com/apache/hadoop-ozone/pull/1328 ## What changes were proposed in this pull request? When ozone.om.enable.filesystem.paths, OM normalizes path, and stores the Keyname. Now when user tries to read the file from S3 using the keyName which user has used to create the Key, it will return error KEY_NOT_FOUND The issue is, lookupKey also need to normalize path, when ozone.om.enable.filesystem.paths is enabled. This is a common API used by S3/FS. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4102 ## How was this patch tested? Added a test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 closed pull request #1315: HDDS-4102. Normalize Keypath for lookupKey.
bharatviswa504 closed pull request #1315: URL: https://github.com/apache/hadoop-ozone/pull/1315 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4116) SCM CLI command towards certain IP
[ https://issues.apache.org/jira/browse/HDDS-4116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177192#comment-17177192 ] Attila Doroszlai commented on HDDS-4116: [~timmylicheng], can you please describe what's the goal of this task? > SCM CLI command towards certain IP > -- > > Key: HDDS-4116 > URL: https://issues.apache.org/jira/browse/HDDS-4116 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Li Cheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] codecov-commenter commented on pull request #1327: HDDS-4098. Improve om admin getserviceroles error message
codecov-commenter commented on pull request #1327: URL: https://github.com/apache/hadoop-ozone/pull/1327#issuecomment-673605022 # [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1327?src=pr=h1) Report > Merging [#1327](https://codecov.io/gh/apache/hadoop-ozone/pull/1327?src=pr=desc) into [master](https://codecov.io/gh/apache/hadoop-ozone/commit/9c2218047bf8cf13f9c52ad819c938d9ba964b81=desc) will **decrease** coverage by `0.07%`. > The diff coverage is `50.00%`. [](https://codecov.io/gh/apache/hadoop-ozone/pull/1327?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#1327 +/- ## - Coverage 74.41% 74.34% -0.08% - Complexity1030110319 +18 Files 984 985 +1 Lines 5039450534 +140 Branches 4914 4934 +20 + Hits 3750337568 +65 - Misses1053110602 +71 - Partials 2360 2364 +4 ``` | [Impacted Files](https://codecov.io/gh/apache/hadoop-ozone/pull/1327?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...java/org/apache/hadoop/ozone/admin/om/OMAdmin.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1327/diff?src=pr=tree#diff-aGFkb29wLW96b25lL3Rvb2xzL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3Avb3pvbmUvYWRtaW4vb20vT01BZG1pbi5qYXZh) | `38.09% <50.00%> (ø)` | `3.00 <0.00> (ø)` | | | [...doop/ozone/om/exceptions/OMNotLeaderException.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1327/diff?src=pr=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL29tL2V4Y2VwdGlvbnMvT01Ob3RMZWFkZXJFeGNlcHRpb24uamF2YQ==) | `48.48% <0.00%> (-18.19%)` | `3.00% <0.00%> (-2.00%)` | | | [...hdds/scm/container/common/helpers/ExcludeList.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1327/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy9zY20vY29udGFpbmVyL2NvbW1vbi9oZWxwZXJzL0V4Y2x1ZGVMaXN0LmphdmE=) | `86.95% <0.00%> (-13.05%)` | `19.00% <0.00%> (-3.00%)` | | | [...e/hadoop/ozone/recon/tasks/OMDBUpdatesHandler.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1327/diff?src=pr=tree#diff-aGFkb29wLW96b25lL3JlY29uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3Avb3pvbmUvcmVjb24vdGFza3MvT01EQlVwZGF0ZXNIYW5kbGVyLmphdmE=) | `60.93% <0.00%> (-9.66%)` | `9.00% <0.00%> (-8.00%)` | | | [...ache/hadoop/ozone/om/codec/S3SecretValueCodec.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1327/diff?src=pr=tree#diff-aGFkb29wLW96b25lL296b25lLW1hbmFnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9vbS9jb2RlYy9TM1NlY3JldFZhbHVlQ29kZWMuamF2YQ==) | `90.90% <0.00%> (-9.10%)` | `3.00% <0.00%> (-1.00%)` | | | [...va/org/apache/hadoop/hdds/utils/db/RDBMetrics.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1327/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvZnJhbWV3b3JrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy91dGlscy9kYi9SREJNZXRyaWNzLmphdmE=) | `92.85% <0.00%> (-7.15%)` | `14.00% <0.00%> (-1.00%)` | | | [...p/ozone/security/OzoneDelegationTokenSelector.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1327/diff?src=pr=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL3NlY3VyaXR5L096b25lRGVsZWdhdGlvblRva2VuU2VsZWN0b3IuamF2YQ==) | `72.22% <0.00%> (-5.56%)` | `7.00% <0.00%> (-1.00%)` | | | [.../transport/server/ratis/ContainerStateMachine.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1327/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvY29tbW9uL3RyYW5zcG9ydC9zZXJ2ZXIvcmF0aXMvQ29udGFpbmVyU3RhdGVNYWNoaW5lLmphdmE=) | `71.07% <0.00%> (-5.39%)` | `62.00% <0.00%> (-4.00%)` | | | [...ozone/container/ozoneimpl/ContainerController.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1327/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvb3pvbmVpbXBsL0NvbnRhaW5lckNvbnRyb2xsZXIuamF2YQ==) | `76.31% <0.00%> (-5.27%)` | `13.00% <0.00%> (-1.00%)` | | | [.../apache/hadoop/ozone/protocolPB/OzonePBHelper.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1327/diff?src=pr=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL3Byb3RvY29sUEIvT3pvbmVQQkhlbHBlci5qYXZh) | `90.00% <0.00%> (-5.00%)` | `6.00% <0.00%> (-1.00%)` | | | ... and [45 more](https://codecov.io/gh/apache/hadoop-ozone/pull/1327/diff?src=pr=tree-more) | | -- [Continue to review full report at
[jira] [Updated] (HDDS-4098) Improve om admin getserviceroles error message
[
https://issues.apache.org/jira/browse/HDDS-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-4098:
-
Labels: newbie pull-request-available (was: newbie)
> Improve om admin getserviceroles error message
> --
>
> Key: HDDS-4098
> URL: https://issues.apache.org/jira/browse/HDDS-4098
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone CLI
>Reporter: Attila Doroszlai
>Assignee: Dharmendra Shavkani
>Priority: Minor
> Labels: newbie, pull-request-available
>
> Steps to reproduce:
> # Start sample docker cluster
> # Run {{ozone admin om getserviceroles}} with unknown service ID
> {code:title=repro}
> $ cd hadoop-ozone/dist/target/ozone-*/compose/ozone
> $ docker-compose up -d
> $ docker-compose exec scm bash
> bash-4.2$ ozone admin om getserviceroles --service-id=om
> Error: This command works only on OzoneManager HA cluster. Service ID
> specified does not match with ozone.om.service.ids defined in the
> configuration. Configured ozone.om.service.ids are[]bash-4.2$
> {code}
> * The message should include a space before {{[]}}, and a newline at the end
> (prompt should appear in next line).
> * Wording of the message could also be improved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dshavkanihort opened a new pull request #1327: HDDS-4098. Improve om admin getserviceroles error message
dshavkanihort opened a new pull request #1327: URL: https://github.com/apache/hadoop-ozone/pull/1327 ## What changes were proposed in this pull request? Improve the Error Messages in Display. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4098?filter=-1 ## How was this patch tested? $ cd hadoop-ozone/dist/target/ozone-*/compose/ozone $ docker-compose up -d $ docker-compose exec scm bash bash-4.2$ ozone admin om getserviceroles --service-id=om Error: This command works only on OzoneManager HA cluster. Service ID specified does not match with ozone.om.service.ids defined in the configuration. Configured ozone.om.service.ids are [] bash-4.2$ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4097) S3/Ozone Filesystem inter-op
[
https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177116#comment-17177116
]
Bharat Viswanadham commented on HDDS-4097:
--
Hi [~elek]
Thank You for reviewing design doc.
{quote}1. What does it mean from compatibility point of view? Will it work
exactly the same way as Amazon S3? Does it mean that we start to support a
different semantic when ozone.om.enable.filesystem.paths is turned on?{quote}
Yes, when ozone.om.enable.filesystem.paths, paths are treated as filesystem
paths, so we check file system semantics and normalize the path.
{quote}2. What is the long term plan, why the configuration property is needed?
I mean why can't we change the default behavior? Is there any disadvantage of
this approach?{quote}
This is to provide someone who want to use ozone as just an object store and
does not require any of these semantics to be followed. When the user plans to
choose to ingest via S3, and use via Ozone filesystem this config will help.
(And also planning to make this bucket level property, instead of cluster-wide,
not yet finalized)
{quote}3. Related to 1 + 2. Is it possible to create the intermediate "dir"
keys but remove them from the list when listed from S3?{quote}
Yes, it can be. But right now when this property is enabled, we show all
intermediate directories also. [~arp] brought a point that if we don;t show
intermediate keys, and when user tries to create a key with that intermediate
path it will fail, and the user will be confused intermediate paths are not
shown, and the user is not able to create a key.
Example:
/a/b/c/file1 -> Key create
/a/b/c -> Key Create - fails (As there is already a path with that name exists
as directory) (List if it does not show, the user might be confused, to be
explicit if intermediate paths are shown this would be clear)
>From usability point of view, we can show intermediate dirs. Do you see any
>advantage or any other favorable points in hiding those when list operation?
>We can revisit this if required.
{quote}4. How does it relate to the prefix table approach?{quote}
Not sure, what is meant here. Any more info will help to answer the question.
> S3/Ozone Filesystem inter-op
>
>
> Key: HDDS-4097
> URL: https://issues.apache.org/jira/browse/HDDS-4097
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem
> path enabled.xlsx
>
>
> This Jira is to implement changes required to use Ozone buckets when data is
> ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial
> implementation for this is done as part of HDDS-3955. There are few API's
> which have missed the changes during the implementation of HDDS-3955.
> Attached design document which discusses each API, and what changes are
> required.
> Excel sheet has information about each API, from what all interfaces the OM
> API is used, and what changes are required for the API to support
> inter-operability.
> Note: The proposal for delete/rename is still under discussion, not yet
> finalized.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1315: HDDS-4102. Normalize Keypath for lookupKey.
elek commented on pull request #1315: URL: https://github.com/apache/hadoop-ozone/pull/1315#issuecomment-673536731 Thanks the patch @bharatviswa504. Cross-linking my generic questions: https://issues.apache.org/jira/browse/HDDS-4097 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4097) S3/Ozone Filesystem inter-op
[ https://issues.apache.org/jira/browse/HDDS-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177093#comment-17177093 ] Marton Elek commented on HDDS-4097: --- Thanks to take care bout this @bharatviswa504 I have a few questions about the design, uploaded to the https://issues.apache.org/jira/browse/HDDS-4097. Not sure where is the right place to ask, here or the PR of https://issues.apache.org/jira/browse/HDDS-4102. Overall it looks good to me, but I am trying to understand the vision. 1. What does it mean from compatibility point of view? Will it work exactly the same way as Amazon S3? Does it mean that we start to support a different semantic when ozone.om.enable.filesystem.paths is turned on? 2. What is the long term plan, why the configuration property is needed? I mean why can't we change the default behavior? Is there any disadvantage of this approach? 3. Related to 1 + 2. Is it possible to create the intermediate "dir" keys but remove them from the list when listed from S3? 4. How does it relate to the prefix table approach? > S3/Ozone Filesystem inter-op > > > Key: HDDS-4097 > URL: https://issues.apache.org/jira/browse/HDDS-4097 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: Ozone FileSystem Paths Enabled.docx, Ozone filesystem > path enabled.xlsx > > > This Jira is to implement changes required to use Ozone buckets when data is > ingested via S3 and use the bucket/volume via OzoneFileSystem. Initial > implementation for this is done as part of HDDS-3955. There are few API's > which have missed the changes during the implementation of HDDS-3955. > Attached design document which discusses each API, and what changes are > required. > Excel sheet has information about each API, from what all interfaces the OM > API is used, and what changes are required for the API to support > inter-operability. > Note: The proposal for delete/rename is still under discussion, not yet > finalized. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel commented on pull request #1291: HDDS-4062. Non rack aware pipelines should not be created if multiple racks are alive
sodonnel commented on pull request #1291: URL: https://github.com/apache/hadoop-ozone/pull/1291#issuecomment-673520331 Thanks for the review @fapifta - I will have a look at the test and spilt them into two as you said. I guess I just did it this way as other tests are like this, and there is a reasonable amount of setup code. I do see your point on each test covering only one thing however. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel commented on a change in pull request #1291: HDDS-4062. Non rack aware pipelines should not be created if multiple racks are alive
sodonnel commented on a change in pull request #1291:
URL: https://github.com/apache/hadoop-ozone/pull/1291#discussion_r470003462
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelinePlacementPolicy.java
##
@@ -163,9 +163,42 @@ int currentPipelineCount(DatanodeDetails datanodeDetails,
int nodesRequired) {
throw new SCMException(msg,
SCMException.ResultCodes.FAILED_TO_FIND_SUITABLE_NODE);
}
+
+if (!checkAllNodesAreEqual(nodeManager.getClusterNetworkTopologyMap())) {
+ boolean multipleRacks = multipleRacksAvailable(healthyNodes);
Review comment:
In the first call to `multipleRacksAvailable(...)` we just use all
healthy nodes and don't worry about excluded nodes. As Pifta said, excluded
nodes are handled later and this first call is just to check if the cluster has
multiple alive racks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4107) replace scmID with clusterID for container and volume at Datanode side
[
https://issues.apache.org/jira/browse/HDDS-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177005#comment-17177005
]
Marton Elek commented on HDDS-4107:
---
cc [~avijayan] who is working on the upgrade design
> replace scmID with clusterID for container and volume at Datanode side
> --
>
> Key: HDDS-4107
> URL: https://issues.apache.org/jira/browse/HDDS-4107
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: SCM
>Reporter: Glen Geng
>Assignee: Glen Geng
>Priority: Major
> Labels: backward-incompatible, pull-request-available, upgrade
>
> The disk layout per volume is as follows:
> {code:java}
> ../hdds/VERSION
> ../hdds/<>/current/<>/<>/metadata
> ../hdds/<>/current/<>/<>/<>{code}
> However, after SCM-HA is enabled, a typical SCM group will consists of 3
> SCMs, each of the SCMs has its own scmUuid, meanwhile share the same
> clusterID.
> Since federation is not supported yet, only one cluster is supported now,
> this Jira will change scmID to clusterID for container and volume at Datanode
> side.
> The disk layout after the change will be as follows:
> {code:java}
> ../hdds/VERSION
> ../hdds/<>/current/<>/<>/metadata
> ../hdds/<>/current/<>/<>/<>{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on pull request #1326: HDDS-3441. Enable TestKeyManagerImpl test cases.
aryangupta1998 commented on pull request #1326: URL: https://github.com/apache/hadoop-ozone/pull/1326#issuecomment-673458037 Hi @xiaoyuyao Can you please review this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3441) Enable TestKeyManagerImpl test cases
[ https://issues.apache.org/jira/browse/HDDS-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3441: - Labels: pull-request-available (was: ) > Enable TestKeyManagerImpl test cases > > > Key: HDDS-3441 > URL: https://issues.apache.org/jira/browse/HDDS-3441 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: test >Affects Versions: 0.5.0 >Reporter: Nanda kumar >Assignee: Aryan Gupta >Priority: Major > Labels: pull-request-available > > Fix and enable TestKeyManagerImpl test cases -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 opened a new pull request #1326: HDDS-3441. Enable TestKeyManagerImpl test cases.
aryangupta1998 opened a new pull request #1326: URL: https://github.com/apache/hadoop-ozone/pull/1326 ## What changes were proposed in this pull request? Fixed testCheckAccessForFileKey(). ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3441 ## How was this patch tested? Tested Manually. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4116) SCM CLI command towards certain IP
Li Cheng created HDDS-4116: -- Summary: SCM CLI command towards certain IP Key: HDDS-4116 URL: https://issues.apache.org/jira/browse/HDDS-4116 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Li Cheng -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4115) CLI command to show current SCM leader and follower status
Li Cheng created HDDS-4115: -- Summary: CLI command to show current SCM leader and follower status Key: HDDS-4115 URL: https://issues.apache.org/jira/browse/HDDS-4115 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Li Cheng -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3441) Enable TestKeyManagerImpl test cases
[ https://issues.apache.org/jira/browse/HDDS-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aryan Gupta reassigned HDDS-3441: - Assignee: Aryan Gupta > Enable TestKeyManagerImpl test cases > > > Key: HDDS-3441 > URL: https://issues.apache.org/jira/browse/HDDS-3441 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: test >Affects Versions: 0.5.0 >Reporter: Nanda kumar >Assignee: Aryan Gupta >Priority: Major > > Fix and enable TestKeyManagerImpl test cases -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] codecov-commenter commented on pull request #1325: HDDS-4114. Bump log4j2 version
codecov-commenter commented on pull request #1325: URL: https://github.com/apache/hadoop-ozone/pull/1325#issuecomment-673419156 # [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1325?src=pr=h1) Report > Merging [#1325](https://codecov.io/gh/apache/hadoop-ozone/pull/1325?src=pr=desc) into [master](https://codecov.io/gh/apache/hadoop-ozone/commit/eb70d9ee8482f5e2deb4dabef0d801b147dc980f=desc) will **decrease** coverage by `0.13%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/hadoop-ozone/pull/1325?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#1325 +/- ## - Coverage 74.46% 74.32% -0.14% + Complexity1034010313 -27 Files 985 985 Lines 5053450534 Branches 4934 4934 - Hits 3763137561 -70 - Misses1054610605 +59 - Partials 2357 2368 +11 ``` | [Impacted Files](https://codecov.io/gh/apache/hadoop-ozone/pull/1325?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../java/org/apache/hadoop/ozone/debug/DBScanner.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1325/diff?src=pr=tree#diff-aGFkb29wLW96b25lL3Rvb2xzL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3Avb3pvbmUvZGVidWcvREJTY2FubmVyLmphdmE=) | `2.81% <ø> (ø)` | `2.00 <0.00> (ø)` | | | [.../java/org/apache/hadoop/ozone/debug/RDBParser.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1325/diff?src=pr=tree#diff-aGFkb29wLW96b25lL3Rvb2xzL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3Avb3pvbmUvZGVidWcvUkRCUGFyc2VyLmphdmE=) | `40.00% <ø> (ø)` | `2.00 <0.00> (ø)` | | | [...va/org/apache/hadoop/hdds/utils/db/RDBMetrics.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1325/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvZnJhbWV3b3JrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy91dGlscy9kYi9SREJNZXRyaWNzLmphdmE=) | `85.71% <0.00%> (-14.29%)` | `13.00% <0.00%> (-2.00%)` | | | [...ache/hadoop/ozone/om/codec/S3SecretValueCodec.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1325/diff?src=pr=tree#diff-aGFkb29wLW96b25lL296b25lLW1hbmFnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9vbS9jb2RlYy9TM1NlY3JldFZhbHVlQ29kZWMuamF2YQ==) | `90.90% <0.00%> (-9.10%)` | `3.00% <0.00%> (-1.00%)` | | | [...hdds/scm/container/common/helpers/ExcludeList.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1325/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy9zY20vY29udGFpbmVyL2NvbW1vbi9oZWxwZXJzL0V4Y2x1ZGVMaXN0LmphdmE=) | `78.26% <0.00%> (-8.70%)` | `17.00% <0.00%> (-2.00%)` | | | [...java/org/apache/hadoop/hdds/utils/db/RDBTable.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1325/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvZnJhbWV3b3JrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3AvaGRkcy91dGlscy9kYi9SREJUYWJsZS5qYXZh) | `56.57% <0.00%> (-6.58%)` | `19.00% <0.00%> (-3.00%)` | | | [...e/commandhandler/CreatePipelineCommandHandler.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1325/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvY29tbW9uL3N0YXRlbWFjaGluZS9jb21tYW5kaGFuZGxlci9DcmVhdGVQaXBlbGluZUNvbW1hbmRIYW5kbGVyLmphdmE=) | `81.25% <0.00%> (-6.25%)` | `8.00% <0.00%> (ø%)` | | | [.../transport/server/ratis/ContainerStateMachine.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1325/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvY29tbW9uL3RyYW5zcG9ydC9zZXJ2ZXIvcmF0aXMvQ29udGFpbmVyU3RhdGVNYWNoaW5lLmphdmE=) | `71.07% <0.00%> (-5.61%)` | `62.00% <0.00%> (-5.00%)` | | | [...ozone/container/ozoneimpl/ContainerController.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1325/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvY29udGFpbmVyLXNlcnZpY2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9jb250YWluZXIvb3pvbmVpbXBsL0NvbnRhaW5lckNvbnRyb2xsZXIuamF2YQ==) | `76.31% <0.00%> (-5.27%)` | `13.00% <0.00%> (-1.00%)` | | | [.../apache/hadoop/ozone/protocolPB/OzonePBHelper.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1325/diff?src=pr=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL3Byb3RvY29sUEIvT3pvbmVQQkhlbHBlci5qYXZh) | `90.00% <0.00%> (-5.00%)` | `6.00% <0.00%> (-1.00%)` | | | ... and [12 more](https://codecov.io/gh/apache/hadoop-ozone/pull/1325/diff?src=pr=tree-more) | | -- [Continue to review full report at
[jira] [Updated] (HDDS-4114) Bump log4j2 version
[ https://issues.apache.org/jira/browse/HDDS-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4114: - Labels: pull-request-available (was: ) > Bump log4j2 version > --- > > Key: HDDS-4114 > URL: https://issues.apache.org/jira/browse/HDDS-4114 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > > There are bunch of bugfixes and improvements since the used 2.10: > https://logging.apache.org/log4j/2.x/changes-report.html#a2.13.3 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek opened a new pull request #1325: HDDS-4114. Bump log4j2 version
elek opened a new pull request #1325: URL: https://github.com/apache/hadoop-ozone/pull/1325 ## What changes were proposed in this pull request? There are a bunch of bugfixes and improvements since the used 2.10: https://logging.apache.org/log4j/2.x/changes-report.html#a2.13.3 ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4114 ## How was this patch tested? Full CI: https://github.com/elek/hadoop-ozone/actions/runs/206787922 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong closed pull request #984: HDDS-3654. Let backgroundCreator create pipeline for the support replication factors alternately
maobaolong closed pull request #984: URL: https://github.com/apache/hadoop-ozone/pull/984 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4114) Bump log4j2 version
Marton Elek created HDDS-4114: - Summary: Bump log4j2 version Key: HDDS-4114 URL: https://issues.apache.org/jira/browse/HDDS-4114 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Marton Elek Assignee: Marton Elek There are bunch of bugfixes and improvements since the used 2.10: https://logging.apache.org/log4j/2.x/changes-report.html#a2.13.3 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4107) replace scmID with clusterID for container and volume at Datanode side
[
https://issues.apache.org/jira/browse/HDDS-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDDS-4107:
Labels: backward-incompatible pull-request-available upgrade (was:
pull-request-available)
> replace scmID with clusterID for container and volume at Datanode side
> --
>
> Key: HDDS-4107
> URL: https://issues.apache.org/jira/browse/HDDS-4107
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: SCM
>Reporter: Glen Geng
>Assignee: Glen Geng
>Priority: Major
> Labels: backward-incompatible, pull-request-available, upgrade
>
> The disk layout per volume is as follows:
> {code:java}
> ../hdds/VERSION
> ../hdds/<>/current/<>/<>/metadata
> ../hdds/<>/current/<>/<>/<>{code}
> However, after SCM-HA is enabled, a typical SCM group will consists of 3
> SCMs, each of the SCMs has its own scmUuid, meanwhile share the same
> clusterID.
> Since federation is not supported yet, only one cluster is supported now,
> this Jira will change scmID to clusterID for container and volume at Datanode
> side.
> The disk layout after the change will be as follows:
> {code:java}
> ../hdds/VERSION
> ../hdds/<>/current/<>/<>/metadata
> ../hdds/<>/current/<>/<>/<>{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4037) Incorrect container numberOfKeys and usedBytes in SCM after key deletetion
[
https://issues.apache.org/jira/browse/HDDS-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen resolved HDDS-4037.
--
Fix Version/s: 0.7.0
Resolution: Fixed
> Incorrect container numberOfKeys and usedBytes in SCM after key deletetion
> --
>
> Key: HDDS-4037
> URL: https://issues.apache.org/jira/browse/HDDS-4037
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Assignee: Sammi Chen
>Priority: Critical
> Labels: pull-request-available
> Fix For: 0.7.0
>
>
> bin/ozone admin container list -s=3 -c=1
> {
> "state" : "CLOSED",
> "replicationFactor" : "THREE",
> "replicationType" : "RATIS",
> "usedBytes" : 2153526137,
> "numberOfKeys" : 57,
> "lastUsed" : "2020-07-27T13:37:16.305Z",
> "stateEnterTime" : "1970-01-10T08:12:26.645Z",
> "owner" : "a46123a8-be63-4736-9478-ce4d8ac845cc",
> "containerID" : 4,
> "deleteTransactionId" : 799549,
> "sequenceId" : 0,
> "open" : false
> }
> `-- 4
> | |-- chunks
> | `-- metadata
> | |-- 4-dn-container.db
> | | |-- 52.sst
> | | |-- 000121.log
> | | |-- CURRENT
> | | |-- IDENTITY
> | | |-- LOCK
> | | |-- LOG
> | | |-- LOG.old.1574862506326174
> | | |-- LOG.old.1575885724469410
> | | |-- LOG.old.1579433296221409
> | | |-- LOG.old.1583661885934875
> | | |-- LOG.old.1583669077602829
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr removed a comment on pull request #1315: HDDS-4102. Normalize Keypath for lookupKey.
rakeshadr removed a comment on pull request #1315: URL: https://github.com/apache/hadoop-ozone/pull/1315#issuecomment-673281343 Thanks @bharatviswa504 for the contribution. TestOzoneFileSystem#testDeleteCreatesFakeParentDir test is failing with this fix. Please take care this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4102) Normalize Keypath for lookupKey
[ https://issues.apache.org/jira/browse/HDDS-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4102: - Labels: pull-request-available (was: ) > Normalize Keypath for lookupKey > --- > > Key: HDDS-4102 > URL: https://issues.apache.org/jira/browse/HDDS-4102 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > When ozone.om.enable.filesystem.paths, OM normalizes path, and stores the > Keyname. > Now when user tries to read the file from S3 using the keyName which user has > used to create the Key, it will return error KEY_NOT_FOUND > The issue is, lookupKey need to normalize path, when > ozone.om.enable.filesystem.paths is enabled. This is common API used by > S3/FS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on pull request #1315: HDDS-4102. Normalize Keypath for lookupKey.
rakeshadr commented on pull request #1315: URL: https://github.com/apache/hadoop-ozone/pull/1315#issuecomment-673281343 Thanks @bharatviswa504 for the contribution. TestOzoneFileSystem#testDeleteCreatesFakeParentDir test is failing with this fix. Please take care this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
