[jira] [Created] (HDFS-17197) Show file replication when listing corrupt files.

2023-09-17 Thread Shuyan Zhang (Jira) 
Shuyan Zhang created HDFS-17197:
---

 Summary: Show file replication when listing corrupt files.
 Key: HDFS-17197
 URL: https://issues.apache.org/jira/browse/HDFS-17197
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Shuyan Zhang


Files with different replication have different reliability guarantees. We need 
to pay attention to corrupted files with a specified replication greater than 
or equal to 3. So, when listing corrupt files, it would be useful to display 
the corresponding replication of the files.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-17197) Show file replication when listing corrupt files.

2023-09-17 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766068#comment-17766068
 ] 

ASF GitHub Bot commented on HDFS-17197:
---

zhangshuyan0 opened a new pull request, #6095:
URL: https://github.com/apache/hadoop/pull/6095

   ### Description of PR
   Files with different replication have different reliability guarantees. We 
need to pay attention to corrupted files with a specified replication greater 
than or equal to 3. So, when listing corrupt files, it would be useful to 
display the corresponding replication of the files.




> Show file replication when listing corrupt files.
> -
>
> Key: HDFS-17197
> URL: https://issues.apache.org/jira/browse/HDFS-17197
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Shuyan Zhang
>Priority: Major
>
> Files with different replication have different reliability guarantees. We 
> need to pay attention to corrupted files with a specified replication greater 
> than or equal to 3. So, when listing corrupt files, it would be useful to 
> display the corresponding replication of the files.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-17197) Show file replication when listing corrupt files.

2023-09-17 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDFS-17197:
--
Labels: pull-request-available  (was: )

> Show file replication when listing corrupt files.
> -
>
> Key: HDFS-17197
> URL: https://issues.apache.org/jira/browse/HDFS-17197
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Shuyan Zhang
>Priority: Major
>  Labels: pull-request-available
>
> Files with different replication have different reliability guarantees. We 
> need to pay attention to corrupted files with a specified replication greater 
> than or equal to 3. So, when listing corrupt files, it would be useful to 
> display the corresponding replication of the files.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-17194) Enhance the log message for striped block recovery

2023-09-17 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766069#comment-17766069
 ] 

ASF GitHub Bot commented on HDFS-17194:
---

hadoop-yetus commented on PR #6094:
URL: https://github.com/apache/hadoop/pull/6094#issuecomment-1722441749

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 26s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include 
any new or modified tests. Please justify why no new tests are needed for this 
patch. Also please list what manual steps were performed to verify this patch.  
|
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  31m 44s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 53s |  |  trunk passed with JDK 
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04  |
   | +1 :green_heart: |  compile  |   0m 46s |  |  trunk passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  checkstyle  |   0m 46s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 54s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 50s |  |  trunk passed with JDK 
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m  9s |  |  trunk passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  spotbugs  |   1m 56s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  21m 53s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 44s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 45s |  |  the patch passed with JDK 
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04  |
   | +1 :green_heart: |  javac  |   0m 45s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 40s |  |  the patch passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  javac  |   0m 40s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 34s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   0m 42s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 38s |  |  the patch passed with JDK 
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m  3s |  |  the patch passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  spotbugs  |   1m 50s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  21m 44s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  | 192m 34s |  |  hadoop-hdfs in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   0m 41s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 284m 29s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6094/2/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/6094 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
   | uname | Linux c43a5ab2308b 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 040ca48885a1a7965a868fd6bcc2ee0c06f97db5 |
   | Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6094/2/testReport/ |
   | Max. process+thread count | 3552 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
   | Console output | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6094/2/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |

[jira] [Commented] (HDFS-17194) Enhance the log message for striped block recovery

2023-09-17 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766073#comment-17766073
 ] 

ASF GitHub Bot commented on HDFS-17194:
---

hadoop-yetus commented on PR #6094:
URL: https://github.com/apache/hadoop/pull/6094#issuecomment-1722453433

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |  12m 51s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include 
any new or modified tests. Please justify why no new tests are needed for this 
patch. Also please list what manual steps were performed to verify this patch.  
|
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  43m 47s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 30s |  |  trunk passed with JDK 
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |   1m 21s |  |  trunk passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  checkstyle  |   1m 12s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 29s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m 12s |  |  trunk passed with JDK 
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m 41s |  |  trunk passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  spotbugs  |   3m 19s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  36m 15s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 14s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 16s |  |  the patch passed with JDK 
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |   1m 16s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 10s |  |  the patch passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  javac  |   1m 10s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  1s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 59s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 16s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 57s |  |  the patch passed with JDK 
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m 29s |  |  the patch passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  spotbugs  |   3m 11s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  35m 51s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  | 224m 41s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6094/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 57s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 378m 48s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6094/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/6094 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
   | uname | Linux de2d5fc27e47 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / bdc7fbe8d5245afb7b94fc3050c74f2d83824dca |
   | Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6094/1/testReport/ |
   | Max. process+thread count | 3442 (vs. ulimit of 5500) |
   |

[jira] [Commented] (HDFS-17197) Show file replication when listing corrupt files.

2023-09-17 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766101#comment-17766101
 ] 

ASF GitHub Bot commented on HDFS-17197:
---

hadoop-yetus commented on PR #6095:
URL: https://github.com/apache/hadoop/pull/6095#issuecomment-1722504685

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 37s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include 
any new or modified tests. Please justify why no new tests are needed for this 
patch. Also please list what manual steps were performed to verify this patch.  
|
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  44m  7s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 24s |  |  trunk passed with JDK 
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |   1m 20s |  |  trunk passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  checkstyle  |   1m 14s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 29s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m 10s |  |  trunk passed with JDK 
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m 41s |  |  trunk passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  spotbugs  |   3m 21s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  35m 54s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 12s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 15s |  |  the patch passed with JDK 
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |   1m 15s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m  9s |  |  the patch passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  javac  |   1m  9s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   1m  1s | 
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6095/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 110 unchanged 
- 1 fixed = 112 total (was 111)  |
   | +1 :green_heart: |  mvnsite  |   1m 17s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 57s |  |  the patch passed with JDK 
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m 31s |  |  the patch passed with JDK 
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05  |
   | +1 :green_heart: |  spotbugs  |   3m 16s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  36m 14s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  | 218m 41s |  |  hadoop-hdfs in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   0m 56s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 360m 38s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.43 ServerAPI=1.43 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6095/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/6095 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
   | uname | Linux 846959e2d4ab 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / b6957d74806004b85674016af809acd72c32c42c |
   | Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6095/1/testReport/ |
   | Max. process+thread count | 3337 (vs. ulimit of 5500) |
   | modules | C: ha

[jira] [Commented] (HDFS-17190) EC: Fix bug of OIV processing XAttr.

2023-09-17 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-17190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766178#comment-17766178
 ] 

ASF GitHub Bot commented on HDFS-17190:
---

Hexiaoqiao merged PR #6067:
URL: https://github.com/apache/hadoop/pull/6067




> EC: Fix bug of OIV processing XAttr.
> 
>
> Key: HDFS-17190
> URL: https://issues.apache.org/jira/browse/HDFS-17190
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Shuyan Zhang
>Priority: Major
>  Labels: pull-request-available
>
> When we need to use OIV to print EC information for a directory, 
> `PBImageTextWriter#getErasureCodingPolicyName` will be called. Currently, 
> this method uses `XATTR_ERASURECODING_POLICY.contains(xattr.getName())` to 
> filter and obtain EC XAttr, which is very dangerous. If we have an XAttr 
> whose name happens to be a substring of `hdfs.erasurecoding.policy`, then 
> `getErasureCodingPolicyName` will return the wrong result. Our internal 
> production environment has customized some XAttrs, and this bug caused errors 
> in the parsing results of OIV when using `-ec` option. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-17190) EC: Fix bug of OIV processing XAttr.

2023-09-17 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-17190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766180#comment-17766180
 ] 

ASF GitHub Bot commented on HDFS-17190:
---

Hexiaoqiao commented on PR #6067:
URL: https://github.com/apache/hadoop/pull/6067#issuecomment-1722678846

   Committed to trunk. Thanks @zhangshuyan0 .




> EC: Fix bug of OIV processing XAttr.
> 
>
> Key: HDFS-17190
> URL: https://issues.apache.org/jira/browse/HDFS-17190
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Shuyan Zhang
>Priority: Major
>  Labels: pull-request-available
>
> When we need to use OIV to print EC information for a directory, 
> `PBImageTextWriter#getErasureCodingPolicyName` will be called. Currently, 
> this method uses `XATTR_ERASURECODING_POLICY.contains(xattr.getName())` to 
> filter and obtain EC XAttr, which is very dangerous. If we have an XAttr 
> whose name happens to be a substring of `hdfs.erasurecoding.policy`, then 
> `getErasureCodingPolicyName` will return the wrong result. Our internal 
> production environment has customized some XAttrs, and this bug caused errors 
> in the parsing results of OIV when using `-ec` option. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-17190) EC: Fix bug of OIV processing XAttr.

2023-09-17 Thread Xiaoqiao He (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-17190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoqiao He resolved HDFS-17190.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
 Assignee: Shuyan Zhang
   Resolution: Fixed

> EC: Fix bug of OIV processing XAttr.
> 
>
> Key: HDFS-17190
> URL: https://issues.apache.org/jira/browse/HDFS-17190
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Shuyan Zhang
>Assignee: Shuyan Zhang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> When we need to use OIV to print EC information for a directory, 
> `PBImageTextWriter#getErasureCodingPolicyName` will be called. Currently, 
> this method uses `XATTR_ERASURECODING_POLICY.contains(xattr.getName())` to 
> filter and obtain EC XAttr, which is very dangerous. If we have an XAttr 
> whose name happens to be a substring of `hdfs.erasurecoding.policy`, then 
> `getErasureCodingPolicyName` will return the wrong result. Our internal 
> production environment has customized some XAttrs, and this bug caused errors 
> in the parsing results of OIV when using `-ec` option. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-17138) RBF: We changed the hadoop.security.auth_to_local configuration of one router, the other routers stopped working

2023-09-17 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766181#comment-17766181
 ] 

ASF GitHub Bot commented on HDFS-17138:
---

zhangxiping1 commented on PR #5921:
URL: https://github.com/apache/hadoop/pull/5921#issuecomment-1722681250

   
![image](https://github.com/apache/hadoop/assets/40832063/4ab3c82a-2afc-4830-8674-06a1c32c10a7)
   window 10 environment, the test case runs successfully ，try to build again




> RBF: We changed the hadoop.security.auth_to_local configuration of one 
> router, the other routers stopped working
> 
>
> Key: HDFS-17138
> URL: https://issues.apache.org/jira/browse/HDFS-17138
> Project: Hadoop HDFS
>  Issue Type: Bug
> Environment: hadoop 3.3.0
>Reporter: Xiping Zhang
>Assignee: Xiping Zhang
>Priority: Major
>  Labels: pull-request-available
> Attachments: image-2023-08-02-16-20-34-454.png, 
> image-2023-08-03-10-32-03-457.png
>
>
> other routers  error log：
> !image-2023-08-02-16-20-34-454.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-17197) Show file replication when listing corrupt files.

2023-09-17 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-17197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766183#comment-17766183
 ] 

ASF GitHub Bot commented on HDFS-17197:
---

Hexiaoqiao commented on code in PR #6095:
URL: https://github.com/apache/hadoop/pull/6095#discussion_r1328207545


##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -6195,7 +6197,11 @@ Collection 
listCorruptFileBlocks(String path,
 if (inode != null) {
   String src = inode.getFullPathName();
   if (isParentEntry(src, path)) {
-corruptFiles.add(new CorruptFileBlockInfo(src, blk));
+int repl = 0;
+if (inode.isFile()) {
+  repl = inode.asFile().getFileReplication();

Review Comment:
   What will be return if this inode is one EC file here?



##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -6131,15 +6131,17 @@ void releaseBackupNode(NamenodeRegistration 
registration)
   static class CorruptFileBlockInfo {
 final String path;
 final Block block;
+private final int replication;
 
-public CorruptFileBlockInfo(String p, Block b) {
+CorruptFileBlockInfo(String p, Block b, int r) {
   path = p;
   block = b;
+  replication = r;
 }
 
 @Override
 public String toString() {
-  return block.getBlockName() + "\t" + path;
+  return block.getBlockName() + "\t" + replication + "\t" + path;

Review Comment:
   I am a little worried about the compatibility while upstream dependency with 
this string but we change the format here.





> Show file replication when listing corrupt files.
> -
>
> Key: HDFS-17197
> URL: https://issues.apache.org/jira/browse/HDFS-17197
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Shuyan Zhang
>Priority: Major
>  Labels: pull-request-available
>
> Files with different replication have different reliability guarantees. We 
> need to pay attention to corrupted files with a specified replication greater 
> than or equal to 3. So, when listing corrupt files, it would be useful to 
> display the corresponding replication of the files.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-17129) mis-order of ibr and fbr on datanode

2023-09-17 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-17129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766185#comment-17766185
 ] 

ASF GitHub Bot commented on HDFS-17129:
---

LiuGuH commented on PR #5892:
URL: https://github.com/apache/hadoop/pull/5892#issuecomment-1722705134

   https://github.com/apache/hadoop/pull/6082 is the same code as this.
   In that  is a clean build and Test Results are Successful.

   




> mis-order of ibr and fbr on datanode 
> -
>
> Key: HDFS-17129
> URL: https://issues.apache.org/jira/browse/HDFS-17129
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: hdfs3.4.0
>Reporter: liuguanghua
>Assignee: liuguanghua
>Priority: Major
>  Labels: pull-request-available
>
> HDFS-16016 , provide new thread to handler IBR. That is a greate improvement. 
> But it maybe casue the mis-order of ibr and fbr



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-17198) RBF: fix bug of getRepresentativeQuorum

2023-09-17 Thread Jian Zhang (Jira) 
Jian Zhang created HDFS-17198:
-

 Summary: RBF: fix bug of getRepresentativeQuorum
 Key: HDFS-17198
 URL: https://issues.apache.org/jira/browse/HDFS-17198
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Jian Zhang


h2. *Bug description*

In the original implementation, when each router reports nn status at different 
times, the nn status is the status reported by majority routers, for example:
router1 -> nn0:active dateModified:1

router2 -> nn0:active dateModified:2

router3 -> nn0:active dateModified:3

router0 -> nn0:standby dateModified:4

Then, the status of nn0 is active, because majority routers report that nn0 is 
active.

If majority routers report nn status at the same time, for example:
(record1) router1 -> nn0:active dateModified:1

(record2) router2 -> nn0:active dateModified:1

(record3) router3 -> nn0:active dateModified:1

(record4) router0 -> nn0:standbydateModified:2

Then the state of nn0 is standby, but We expect the status of nn0 is active

This bug is because the above record is put into the Treeset in the method 
getRepresentativeQuorum. Since record1,2,3 have the same dateModified, there 
will only be one record in the final treeset of this method, so this method 
thinks that this nn is standby, because record4 newer
h2. *How to reproduce*

Running my unit test testRegistrationMajorityQuorumEqDateModified, but using 
the original code



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-17198) RBF: fix bug of getRepresentativeQuorum

2023-09-17 Thread Jian Zhang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian Zhang reassigned HDFS-17198:
-

Assignee: Jian Zhang

> RBF: fix bug of getRepresentativeQuorum
> ---
>
> Key: HDFS-17198
> URL: https://issues.apache.org/jira/browse/HDFS-17198
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
>
> h2. *Bug description*
> In the original implementation, when each router reports nn status at 
> different times, the nn status is the status reported by majority routers, 
> for example:
> router1 -> nn0:active dateModified:1
> router2 -> nn0:active dateModified:2
> router3 -> nn0:active dateModified:3
> router0 -> nn0:standby dateModified:4
> Then, the status of nn0 is active, because majority routers report that nn0 
> is active.
> If majority routers report nn status at the same time, for example:
> (record1) router1 -> nn0:active dateModified:1
> (record2) router2 -> nn0:active dateModified:1
> (record3) router3 -> nn0:active dateModified:1
> (record4) router0 -> nn0:standbydateModified:2
> Then the state of nn0 is standby, but We expect the status of nn0 is active
> This bug is because the above record is put into the Treeset in the method 
> getRepresentativeQuorum. Since record1,2,3 have the same dateModified, there 
> will only be one record in the final treeset of this method, so this method 
> thinks that this nn is standby, because record4 newer
> h2. *How to reproduce*
> Running my unit test testRegistrationMajorityQuorumEqDateModified, but using 
> the original code



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-17198) RBF: fix bug of getRepresentativeQuorum when records have same dateModified

2023-09-17 Thread Jian Zhang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian Zhang updated HDFS-17198:
--
Summary: RBF: fix bug of getRepresentativeQuorum when records have same 
dateModified  (was: RBF: fix bug of getRepresentativeQuorum)

> RBF: fix bug of getRepresentativeQuorum when records have same dateModified
> ---
>
> Key: HDFS-17198
> URL: https://issues.apache.org/jira/browse/HDFS-17198
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
>
> h2. *Bug description*
> In the original implementation, when each router reports nn status at 
> different times, the nn status is the status reported by majority routers, 
> for example:
> router1 -> nn0:active dateModified:1
> router2 -> nn0:active dateModified:2
> router3 -> nn0:active dateModified:3
> router0 -> nn0:standby dateModified:4
> Then, the status of nn0 is active, because majority routers report that nn0 
> is active.
> If majority routers report nn status at the same time, for example:
> (record1) router1 -> nn0:active dateModified:1
> (record2) router2 -> nn0:active dateModified:1
> (record3) router3 -> nn0:active dateModified:1
> (record4) router0 -> nn0:standbydateModified:2
> Then the state of nn0 is standby, but We expect the status of nn0 is active
> This bug is because the above record is put into the Treeset in the method 
> getRepresentativeQuorum. Since record1,2,3 have the same dateModified, there 
> will only be one record in the final treeset of this method, so this method 
> thinks that this nn is standby, because record4 newer
> h2. *How to reproduce*
> Running my unit test testRegistrationMajorityQuorumEqDateModified, but using 
> the original code



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-17198) RBF: fix bug of getRepresentativeQuorum when records have same dateModified

2023-09-17 Thread Jian Zhang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian Zhang updated HDFS-17198:
--
Attachment: HDFS-17198.v001.patch
Status: Patch Available  (was: Open)

> RBF: fix bug of getRepresentativeQuorum when records have same dateModified
> ---
>
> Key: HDFS-17198
> URL: https://issues.apache.org/jira/browse/HDFS-17198
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Attachments: HDFS-17198.v001.patch
>
>
> h2. *Bug description*
> In the original implementation, when each router reports nn status at 
> different times, the nn status is the status reported by majority routers, 
> for example:
> router1 -> nn0:active dateModified:1
> router2 -> nn0:active dateModified:2
> router3 -> nn0:active dateModified:3
> router0 -> nn0:standby dateModified:4
> Then, the status of nn0 is active, because majority routers report that nn0 
> is active.
> If majority routers report nn status at the same time, for example:
> (record1) router1 -> nn0:active dateModified:1
> (record2) router2 -> nn0:active dateModified:1
> (record3) router3 -> nn0:active dateModified:1
> (record4) router0 -> nn0:standbydateModified:2
> Then the state of nn0 is standby, but We expect the status of nn0 is active
> This bug is because the above record is put into the Treeset in the method 
> getRepresentativeQuorum. Since record1,2,3 have the same dateModified, there 
> will only be one record in the final treeset of this method, so this method 
> thinks that this nn is standby, because record4 newer
> h2. *How to reproduce*
> Running my unit test testRegistrationMajorityQuorumEqDateModified, but using 
> the original code



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16000) HDFS : Rename performance optimization

2023-09-17 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-16000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766194#comment-17766194
 ] 

ASF GitHub Bot commented on HDFS-16000:
---

Hexiaoqiao commented on code in PR #2964:
URL: https://github.com/apache/hadoop/pull/2964#discussion_r1328244599


##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java:
##
@@ -1468,6 +1475,30 @@ static Collection 
normalizePaths(Collection paths,
 return normalized;
   }
 
+  /**
+   * Get the first Node that sets Quota.
+   */
+  static INode getFirstSetQuotaParentNode(INodesInPath iip) {
+for (int i = iip.length() - 1; i > 0; i--) {
+  INode currNode = iip.getINode(i);
+  if (currNode == null) {

Review Comment:
   Will it meet null here? if it is not expected, we should throw exception IMO.



##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java:
##
@@ -470,17 +475,53 @@ static RenameResult unprotectedRenameTo(FSDirectory fsd,
   }
 } finally {
   if (undoRemoveSrc) {
-tx.restoreSource();
+tx.restoreSource(srcStoragePolicyCounts);
   }
   if (undoRemoveDst) { // Rename failed - restore dst
-tx.restoreDst(bsps);
+tx.restoreDst(bsps, dstStoragePolicyCounts);
   }
 }
 NameNode.stateChangeLog.warn("DIR* FSDirectory.unprotectedRenameTo: " +
 "failed to rename " + src + " to " + dst);
 throw new IOException("rename from " + src + " to " + dst + " failed.");
   }
 
+  /*
+   * Calculate QuotaCounts based on parent directory and storage policy
+   * 1. If the storage policy of src and dst are different,
+   *  calculate the QuotaCounts of src and dst respectively.
+   * 2. If all parent nodes of src and dst are not set with Quota,
+   *  there is no need to calculate QuotaCount.
+   * 3. if parent nodes of src and dst have Quota configured,
+   *  the QuotaCount is calculated once using the storage policy of src.
+   * */
+  private static void computeQuotaCounts(
+  QuotaCounts srcStoragePolicyCounts,
+  QuotaCounts dstStoragePolicyCounts,
+  INodesInPath srcIIP,
+  INodesInPath dstIIP,
+  BlockStoragePolicySuite bsps,
+  RenameOperation tx) {
+INode dstParent = dstIIP.getINode(-2);
+INode srcParentNode = FSDirectory.
+getFirstSetQuotaParentNode(srcIIP);
+INode srcInode = srcIIP.getLastINode();
+INode dstParentNode = FSDirectory.
+getFirstSetQuotaParentNode(dstIIP);
+byte srcStoragePolicyID = FSDirectory.getStoragePolicyId(srcInode);
+byte dstStoragePolicyID = FSDirectory.getStoragePolicyId(dstParent);
+if (srcStoragePolicyID != dstStoragePolicyID) {
+  srcStoragePolicyCounts.add(srcIIP.getLastINode().
+  computeQuotaUsage(bsps));
+  dstStoragePolicyCounts.add(srcIIP.getLastINode()

Review Comment:
   IIUC, this result will be used for the next verify and storage used 
addition/subtraction for src and dst inode, right? But I am confused if it will 
meet some issues here, given directory /a/b (whose storage policy is HDD), /c/d 
(whose storage policy is SSD), when rename from /a/b/r1 (let's assume 1GB used) 
to /c/d/r2, then total HDD storage used will decrease 1GB, and SSD storage used 
increase 1GB, this will be different with fact?



##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java:
##
@@ -470,17 +475,53 @@ static RenameResult unprotectedRenameTo(FSDirectory fsd,
   }
 } finally {
   if (undoRemoveSrc) {
-tx.restoreSource();
+tx.restoreSource(srcStoragePolicyCounts);
   }
   if (undoRemoveDst) { // Rename failed - restore dst
-tx.restoreDst(bsps);
+tx.restoreDst(bsps, dstStoragePolicyCounts);
   }
 }
 NameNode.stateChangeLog.warn("DIR* FSDirectory.unprotectedRenameTo: " +
 "failed to rename " + src + " to " + dst);
 throw new IOException("rename from " + src + " to " + dst + " failed.");
   }
 
+  /*
+   * Calculate QuotaCounts based on parent directory and storage policy
+   * 1. If the storage policy of src and dst are different,
+   *  calculate the QuotaCounts of src and dst respectively.
+   * 2. If all parent nodes of src and dst are not set with Quota,
+   *  there is no need to calculate QuotaCount.
+   * 3. if parent nodes of src and dst have Quota configured,
+   *  the QuotaCount is calculated once using the storage policy of src.
+   * */
+  private static void computeQuotaCounts(
+  QuotaCounts srcStoragePolicyCounts,
+  QuotaCounts dstStoragePolicyCounts,
+  INodesInPath srcIIP,
+  INodesInPath dstIIP,
+  BlockStoragePolicySuite bsps,
+  RenameOperation tx) {
+INode dstParent = dstIIP.getINode(-2);
+