[jira] [Commented] (HDFS-16740) Mini cluster test flakiness

2022-10-20 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17621282#comment-17621282
 ] 

ASF GitHub Bot commented on HDFS-16740:
---

steveloughran commented on code in PR #4835:
URL: https://github.com/apache/hadoop/pull/4835#discussion_r1000928173


##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRollingUpgrade.java:
##
@@ -83,37 +83,18 @@ public static void runCmd(DFSAdmin dfsadmin, boolean 
success,
   }
 
   @Rule
-  public TemporaryFolder folder = new TemporaryFolder();
-
-  /**
-   * Create a default HDFS configuration which has test-specific data 
directories.  This is
-   * intended to protect against interactions between test runs that might 
corrupt results.  Each
-   * test run's data is automatically cleaned-up by JUnit.
-   *
-   * @return a default configuration with test-specific data directories
-   */
-  public Configuration getHdfsConfiguration() throws IOException {
-Configuration conf = new HdfsConfiguration();

Review Comment:
   retain this, but just return the new config. allows for changes later



##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniJournalCluster.java:
##
@@ -64,7 +65,17 @@ public static class Builder {
 public Builder(Configuration conf) {
   this.conf = conf;
 }
-
+
+public Builder(Configuration conf, TemporaryFolder baseDir) {

Review Comment:
   i don't want to add junit dependencies here. better to take a File ref and 
pass it in when used



##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java:
##
@@ -240,6 +241,10 @@ public Builder(Configuration conf) {
   }
 }
 
+public Builder(Configuration conf, TemporaryFolder baseDir) {

Review Comment:
   i don't want to add junit dependencies here; we don't know where else it is 
used. and test dependencies don't get exported by maven. the code will need to 
be given baseDir.getRoot()





> Mini cluster test flakiness
> ---
>
> Key: HDFS-16740
> URL: https://issues.apache.org/jira/browse/HDFS-16740
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs, test
>Affects Versions: 3.4.0, 3.3.5
>Reporter: Steve Vaughan
>Assignee: Steve Vaughan
>Priority: Major
>  Labels: pull-request-available
>
> Mini clusters used during HDFS unit tests are reporting test failures that do 
> not appear to be directly related to submitted changes.  The failures are the 
> result of either interactions between tests run in parallel, or tests which 
> share common disk space for tests.  In all cases, the tests can be run 
> individually serially without any errors.  Addressing this issue will 
> simplify future submissions by eliminating the confusion introduced by these 
> unrelated test failures.
> We can apply lessons recently from TestRollingUpgrade, which was recently 
> patched to unblock a recent submission.  The fixes involved changing the HDFS 
> configuration to use temporary disk space for each individual tests, and 
> using try-with-resources to ensure that clusters were shutdown cleanly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16809) EC striped block is not sufficient when doing in maintenance

2022-10-20 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-16809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17621272#comment-17621272
 ] 

ASF GitHub Bot commented on HDFS-16809:
---

hadoop-yetus commented on PR #5050:
URL: https://github.com/apache/hadoop/pull/5050#issuecomment-1285904505

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m 20s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include 
any new or modified tests. Please justify why no new tests are needed for this 
patch. Also please list what manual steps were performed to verify this patch.  
|
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  43m 42s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 43s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |   1m 28s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 17s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 40s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m 16s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m 40s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   3m 48s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  26m 21s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 27s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 31s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |   1m 31s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 23s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   1m 23s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   0m 58s | 
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5050/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 81 unchanged - 
1 fixed = 82 total (was 82)  |
   | +1 :green_heart: |  mvnsite  |   1m 28s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 56s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m 35s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   3m 40s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  26m 24s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  | 388m 37s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5050/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 58s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 511m  5s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5050/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5050 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
   | uname | Linux 4937a3335e81 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 4745364a75f5b38d6b4a5446f0fcf7c755ccf85e |
   | Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
   | Multi-JDK versions |

[jira] [Resolved] (HDFS-16795) Use secure XML parser utils in hdfs classes

2022-10-20 Thread Steve Loughran (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16795.
---
Fix Version/s: 3.3.5
   Resolution: Fixed

> Use secure XML parser utils in hdfs classes
> ---
>
> Key: HDFS-16795
> URL: https://issues.apache.org/jira/browse/HDFS-16795
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: PJ Fanning
>Assignee: PJ Fanning
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 3.3.9
>
>
> Uptakes HADOOP-18469



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-16807) Improve legacy ClientProtocol#rename2() interface

2022-10-20 Thread JiangHua Zhu (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

JiangHua Zhu updated HDFS-16807:

Affects Version/s: 2.9.2

> Improve legacy ClientProtocol#rename2() interface
> -
>
> Key: HDFS-16807
> URL: https://issues.apache.org/jira/browse/HDFS-16807
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: dfsclient
>Affects Versions: 2.9.2, 3.3.3
>Reporter: JiangHua Zhu
>Priority: Major
>
> In HDFS-2298, rename2() replaced rename(), which is a very meaningful 
> improvement. It looks like some old customs are still preserved, they are:
> 1. When using the shell to execute the mv command, rename() is still used.
> ./bin/hdfs dfs -mv [source] [target]
> {code:java}
> In MoveCommands#Rename:
> protected void processPath(PathData src, PathData target) throws 
> IOException {
>   ..
>   if (!target.fs.rename(src.path, target.path)) {
> // we have no way to know the actual error...
> throw new PathIOException(src.toString());
>   }
> }
> {code}
> 2. When NNThroughputBenchmark verifies the rename.
> In NNThroughputBenchmark#RenameFileStats:
> {code:java}
> long executeOp(int daemonId, int inputIdx, String ignore)
> throws IOException {
>   long start = Time.now();
>   clientProto.rename(fileNames[daemonId][inputIdx],
>   destNames[daemonId][inputIdx]);
>   long end = Time.now();
>   return end-start;
> }
> {code}
> I think the interface should be kept uniform since rename() is deprecated. 
> For NNThroughputBenchmark, it's easy. But it is not easy to improve 
> MoveCommands, because it involves the transformation of FileSystem.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-16795) Use secure XML parser utils in hdfs classes

2022-10-20 Thread Steve Loughran (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran reassigned HDFS-16795:
-

Assignee: PJ Fanning

> Use secure XML parser utils in hdfs classes
> ---
>
> Key: HDFS-16795
> URL: https://issues.apache.org/jira/browse/HDFS-16795
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: PJ Fanning
>Assignee: PJ Fanning
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> Uptakes HADOOP-18469



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-16795) Use secure XML parser utils in hdfs classes

2022-10-20 Thread Steve Loughran (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HDFS-16795:
--
Fix Version/s: 3.4.0
   3.3.9

> Use secure XML parser utils in hdfs classes
> ---
>
> Key: HDFS-16795
> URL: https://issues.apache.org/jira/browse/HDFS-16795
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: PJ Fanning
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> Uptakes HADOOP-18469



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16795) Use secure XML parser utils in hdfs classes

2022-10-20 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17621242#comment-17621242
 ] 

ASF GitHub Bot commented on HDFS-16795:
---

steveloughran merged PR #5016:
URL: https://github.com/apache/hadoop/pull/5016




> Use secure XML parser utils in hdfs classes
> ---
>
> Key: HDFS-16795
> URL: https://issues.apache.org/jira/browse/HDFS-16795
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: PJ Fanning
>Priority: Major
>  Labels: pull-request-available
>
> Uptakes HADOOP-18469



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16809) EC striped block is not sufficient when doing in maintenance

2022-10-20 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-16809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17621084#comment-17621084
 ] 

ASF GitHub Bot commented on HDFS-16809:
---

tasanuma commented on PR #5050:
URL: https://github.com/apache/hadoop/pull/5050#issuecomment-1285472477

   @dingshun3016 Thanks for reporting the issue and submitting the PR. Is it 
possible to add a unit test?




> EC striped block is not sufficient when doing in maintenance
> 
>
> Key: HDFS-16809
> URL: https://issues.apache.org/jira/browse/HDFS-16809
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: ec, hdfs
>Reporter: dingshun
>Priority: Major
>  Labels: pull-request-available
>
> When doing maintenance, ec striped block is not sufficient, which will lead 
> to miss block



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space

2022-10-20 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17621054#comment-17621054
 ] 

ASF GitHub Bot commented on HDFS-3570:
--

ashutoshcipher commented on PR #5044:
URL: https://github.com/apache/hadoop/pull/5044#issuecomment-1285423482

   @aajisaka @ZanderXu - Please help in reviewing . Thanks.




> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used 
> space
> 
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer  mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: pull-request-available
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch, 
> HDFS-3570.aash.1.patch
>
>
> Report from a user here: 
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
>  post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among 
> dfs.data.dirs, and very little DFS usage (which is computed against total 
> possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that 
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a 
> DFS Usage report from DN is 8% only, its got a lot of free space to write 
> more blocks, when that isn't true as shown by the case of this user. It went 
> on scheduling writes to the DN to balance it out, but the DN simply can't 
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on 
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current 
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space 
> being used for non DN data, but it does expose a valid bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16808) HDFS metrics will hold the previous value if there is no new call

2022-10-20 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-16808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620981#comment-17620981
 ] 

ASF GitHub Bot commented on HDFS-16808:
---

hadoop-yetus commented on PR #5049:
URL: https://github.com/apache/hadoop/pull/5049#issuecomment-1285282355

   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 59s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  1s |  |  detect-secrets was not available.  
|
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 1 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  39m  9s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  23m 12s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  compile  |  22m 41s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 33s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   2m  6s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m 30s |  |  trunk passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   0m 59s |  |  trunk passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   3m 13s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  23m 50s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m  4s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  22m 38s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javac  |  22m 38s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m 58s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |  20m 58s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   1m 35s | 
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5049/1/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
 |  hadoop-common-project/hadoop-common: The patch generated 1 new + 10 
unchanged - 0 fixed = 11 total (was 10)  |
   | +1 :green_heart: |  mvnsite  |   1m 55s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   1m 30s |  |  the patch passed with JDK 
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04  |
   | +1 :green_heart: |  javadoc  |   1m 20s |  |  the patch passed with JDK 
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   2m 56s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  23m 10s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  |  18m 43s | 
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5049/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
 |  hadoop-common in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 15s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 217m 15s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.metrics2.lib.TestMutableMetrics |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5049/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/5049 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
   | uname | Linux d3a00330233f 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 99aa7009b558c2a9da8f996766253430857ad5b7 |
   | Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private

[jira] [Updated] (HDFS-16809) EC striped block is not sufficient when doing in maintenance

2022-10-20 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-16809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDFS-16809:
--
Labels: pull-request-available  (was: )

> EC striped block is not sufficient when doing in maintenance
> 
>
> Key: HDFS-16809
> URL: https://issues.apache.org/jira/browse/HDFS-16809
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: ec, hdfs
>Reporter: dingshun
>Priority: Major
>  Labels: pull-request-available
>
> When doing maintenance, ec striped block is not sufficient, which will lead 
> to miss block



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16809) EC striped block is not sufficient when doing in maintenance

2022-10-20 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-16809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620907#comment-17620907
 ] 

ASF GitHub Bot commented on HDFS-16809:
---

dingshun3016 opened a new pull request, #5050:
URL: https://github.com/apache/hadoop/pull/5050

   When doing maintenance, ec striped block is not sufficient, which will lead 
to miss block. 




> EC striped block is not sufficient when doing in maintenance
> 
>
> Key: HDFS-16809
> URL: https://issues.apache.org/jira/browse/HDFS-16809
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: ec, hdfs
>Reporter: dingshun
>Priority: Major
>
> When doing maintenance, ec striped block is not sufficient, which will lead 
> to miss block



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-16809) EC striped block is not sufficient when doing in maintenance

2022-10-20 Thread dingshun (Jira) 
dingshun created HDFS-16809:
---

 Summary: EC striped block is not sufficient when doing in 
maintenance
 Key: HDFS-16809
 URL: https://issues.apache.org/jira/browse/HDFS-16809
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: ec, hdfs
Reporter: dingshun


When doing maintenance, ec striped block is not sufficient, which will lead to 
miss block



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-16808) HDFS metrics will hold the previous value if there is no new call

2022-10-20 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-16808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDFS-16808:
--
Labels: pull-request-available  (was: )

> HDFS metrics will hold the previous value if there is no new call
> -
>
> Key: HDFS-16808
> URL: https://issues.apache.org/jira/browse/HDFS-16808
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Reporter: leo sun
>Priority: Major
>  Labels: pull-request-available
> Attachments: image-2022-10-19-23-59-19-673.png
>
>
> According to the implementation of MutableStat.snapshot(), HDFS metrics will 
> always hold the previous value if there is no more new call.
> It will cause even if user switch active and standby, the previous 
> ANN(standby now) will always output the old value as the pic shows
> !image-2022-10-19-23-59-19-673.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16808) HDFS metrics will hold the previous value if there is no new call

2022-10-20 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/HDFS-16808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620779#comment-17620779
 ] 

ASF GitHub Bot commented on HDFS-16808:
---

ted12138 opened a new pull request, #5049:
URL: https://github.com/apache/hadoop/pull/5049

   
   
   ### Description of PR
   
   
   ### How was this patch tested?
   
   
   ### For code changes:
   
   - [ ] Does the title or this PR starts with the corresponding JIRA issue id 
(e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: have the integration tests been executed and the 
endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, 
`NOTICE-binary` files?
   
   




> HDFS metrics will hold the previous value if there is no new call
> -
>
> Key: HDFS-16808
> URL: https://issues.apache.org/jira/browse/HDFS-16808
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Reporter: leo sun
>Priority: Major
> Attachments: image-2022-10-19-23-59-19-673.png
>
>
> According to the implementation of MutableStat.snapshot(), HDFS metrics will 
> always hold the previous value if there is no more new call.
> It will cause even if user switch active and standby, the previous 
> ANN(standby now) will always output the old value as the pic shows
> !image-2022-10-19-23-59-19-673.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16806) ec data balancer block blk_id The index error ,Data cannot be moved

2022-10-20 Thread ruiliang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HDFS-16806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ruiliang resolved HDFS-16806.
-
Hadoop Flags: Reviewed
  Resolution: Fixed

> ec data balancer block blk_id The index error ,Data cannot be moved
> ---
>
> Key: HDFS-16806
> URL: https://issues.apache.org/jira/browse/HDFS-16806
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 3.1.0
>Reporter: ruiliang
>Priority: Critical
> Attachments: image-2022-10-20-11-32-35-833.png
>
>
> ec data balancer block blk_id The index error ,Data cannot be moved
> dn->10.12.15.149 use disk 100%
>  
> {code:java}
> echo 10.12.15.149>sorucehost
> balancer  -fs hdfs://xxcluster06  -threshold 10 -source -f sorucehost   
> 2>>~/balancer.log &  {code}
>  
> datanode logs 
> A lot of this log output  
> {code:java}
> datanode logs
> ...
> 2022-10-19 14:43:02,031 ERROR datanode.DataNode (DataXceiver.java:run(321)) - 
> fs-hiido-dn-12-15-149.xx.com:1019:DataXceiver error processing COPY_BLOCK 
> operation  src: /10.12.65.216:58214 dst: /10.12.15.149:1019
> org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
> found for 
> BP-1822992414-10.12.65.48-1660893388633:blk_-9223372036799576592_4218617
>         at 
> org.apache.hadoop.hdfs.server.datanode.BlockSender.getReplica(BlockSender.java:492)
>         at 
> org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:256)
>         at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.copyBlock(DataXceiver.java:1089)
>         at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opCopyBlock(Receiver.java:291)
>         at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:113)
>         at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290)
>         at java.lang.Thread.run(Thread.java:748)
> ...    
>     
> hdfs fsck -fs hdfs://xxcluster06 -blockId blk_-9223372036799576592 
> Connecting to namenode via 
> http://fs-hiido-xxcluster06-yynn2.xx.com:50070/fsck?ugi=hdfs=blk_-9223372036799576592+=%2F
> FSCK started by hdfs (auth:KERBEROS_SSL) from /10.12.19.4 at Wed Oct 19 
> 14:47:15 CST 2022Block Id: blk_-9223372036799576592
> Block belongs to: 
> /hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz
> No. of Expected Replica: 5
> No. of live Replica: 5
> No. of excess Replica: 0
> No. of stale Replica: 5
> No. of decommissioned Replica: 0
> No. of decommissioning Replica: 0
> No. of corrupted Replica: 0
> Block replica on datanode/rack: fs-hiido-dn-12-66-4.xx.com/4F08-01-09 is 
> HEALTHY
> Block replica on datanode/rack: fs-hiido-dn-12-65-244.xx.com/4F08-01-08 is 
> HEALTHY
> Block replica on datanode/rack: fs-hiido-dn-12-15-149.xx.com/4F08-05-13 is 
> HEALTHY
> Block replica on datanode/rack: fs-hiido-dn-12-65-218.xx.com/4F08-12-04 is 
> HEALTHY
> Block replica on datanode/rack: fs-hiido-dn-12-17-35.xx.com/4F08-03-03 is 
> HEALTHY
> hdfs fsck -fs hdfs://xxcluster06 
> /hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz
>  -files -blocks -locations
> Connecting to namenode via 
> http://xx.com:50070/fsck?ugi=hdfs=1=1=1=%2Fhive_warehouse%2Fwarehouse_old_snapshots%2Fyy_mbsdkevent_original%2Fdt%3D20210505%2Fpost_202105052129_33.log.gz
> FSCK started by hdfs (auth:KERBEROS_SSL) from /10.12.19.4 for path 
> /hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz
>  at Wed Oct 19 14:48:42 CST 2022
> /hive_warehouse/warehouse_old_snapshots/yy_mbsdkevent_original/dt=20210505/post_202105052129_33.log.gz
>  500582412 bytes, erasure-coded: policy=RS-3-2-1024k, 1 block(s):  OK
> 0. BP-1822992414-10.12.65.48-1660893388633:blk_-9223372036799576592_4218617 
> len=500582412 Live_repl=5  
> [blk_-9223372036799576592:DatanodeInfoWithStorage[10.12.17.35:1019,DS-3ccebf8d-5f05-45b5-ac7f-96d1cfb48608,DISK],
>  
> blk_-9223372036799576591:DatanodeInfoWithStorage[10.12.65.218:1019,DS-4f8e3114-7566-4cf1-ad5a-e454c8ea8805,DISK],
>  
> blk_-9223372036799576590:DatanodeInfoWithStorage[10.12.15.149:1019,DS-1dd55c27-8f47-46a6-935b-1d9024ca9188,DISK],
>  
> blk_-9223372036799576589:DatanodeInfoWithStorage[10.12.65.244:1019,DS-a9ffd747-c427-4aaa-8559-04cded7d9d5f,DISK],
>  
> blk_-9223372036799576588:DatanodeInfoWithStorage[10.12.66.4:1019,DS-d88f94db-6db1-4753-a652-780d7cd7f081,DISK]]
> Status: HEALTHY
>  Number of data-nodes:  62
>  Number of racks:               19
>  Total dirs:                    0
>  Total symlinks:                0Replicated Blocks:
>  Total size:    0 B
>  Total files:   0
>  Total blocks (validated):      0
>  Minimally replicated blocks:   0
>  Over-replicated blocks:        0
>  Under-replicated