[jira] [Work logged] (HDFS-16061) DFTestUtil.waitReplication can produce false positives
[ https://issues.apache.org/jira/browse/HDFS-16061?focusedWorklogId=612353&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612353 ] ASF GitHub Bot logged work on HDFS-16061: - Author: ASF GitHub Bot Created on: 20/Jun/21 05:40 Start Date: 20/Jun/21 05:40 Worklog Time Spent: 10m Work Description: ayushtkn commented on pull request #3095: URL: https://github.com/apache/hadoop/pull/3095#issuecomment-864502846 Thanx @amahussein for the contribution and @jbrennan333 for the review. Sorry, wanted to pull this to lower branches as well, got stuck with some internal stuff, couldn't find time to compile and push. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 612353) Time Spent: 1h (was: 50m) > DFTestUtil.waitReplication can produce false positives > -- > > Key: HDFS-16061 > URL: https://issues.apache.org/jira/browse/HDFS-16061 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > While checking the intermittent failure in > TestBalancerRPCDelay#testBalancerRPCDelayQpsDefault described in HDFS-15146, > I found that the implementation of waitReplication is incorrect. > In the last iteration, when {{correctReplFactor}} is {{false}}, the thread > sleeps for 1 second, then a {{TimeoutException}} is thrown without check > whether the replication was complete in the last second. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16061) DFTestUtil.waitReplication can produce false positives
[ https://issues.apache.org/jira/browse/HDFS-16061?focusedWorklogId=612349&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612349 ] ASF GitHub Bot logged work on HDFS-16061: - Author: ASF GitHub Bot Created on: 20/Jun/21 04:52 Start Date: 20/Jun/21 04:52 Worklog Time Spent: 10m Work Description: ayushtkn merged pull request #3095: URL: https://github.com/apache/hadoop/pull/3095 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 612349) Time Spent: 50m (was: 40m) > DFTestUtil.waitReplication can produce false positives > -- > > Key: HDFS-16061 > URL: https://issues.apache.org/jira/browse/HDFS-16061 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > While checking the intermittent failure in > TestBalancerRPCDelay#testBalancerRPCDelayQpsDefault described in HDFS-15146, > I found that the implementation of waitReplication is incorrect. > In the last iteration, when {{correctReplFactor}} is {{false}}, the thread > sleeps for 1 second, then a {{TimeoutException}} is thrown without check > whether the replication was complete in the last second. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16061) DFTestUtil.waitReplication can produce false positives
[ https://issues.apache.org/jira/browse/HDFS-16061?focusedWorklogId=611759&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-611759 ] ASF GitHub Bot logged work on HDFS-16061: - Author: ASF GitHub Bot Created on: 16/Jun/21 07:09 Start Date: 16/Jun/21 07:09 Worklog Time Spent: 10m Work Description: ayushtkn commented on a change in pull request #3095: URL: https://github.com/apache/hadoop/pull/3095#discussion_r652409047 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java ## @@ -206,6 +206,8 @@ private static final String[] dirNames = { "zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine" }; + + private static final int WAIT_REPLICATION_ATTEMPTS = 40; Review comment: Why you have pulled this up? Means nobody apart from `waitReplication ` method is using it? We could have kept it there itself, the variable is private, no one can change it also -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 611759) Time Spent: 40m (was: 0.5h) > DFTestUtil.waitReplication can produce false positives > -- > > Key: HDFS-16061 > URL: https://issues.apache.org/jira/browse/HDFS-16061 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > While checking the intermittent failure in > TestBalancerRPCDelay#testBalancerRPCDelayQpsDefault described in HDFS-15146, > I found that the implementation of waitReplication is incorrect. > In the last iteration, when {{correctReplFactor}} is {{false}}, the thread > sleeps for 1 second, then a {{TimeoutException}} is thrown without check > whether the replication was complete in the last second. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16061) DFTestUtil.waitReplication can produce false positives
[ https://issues.apache.org/jira/browse/HDFS-16061?focusedWorklogId=610353&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-610353 ] ASF GitHub Bot logged work on HDFS-16061: - Author: ASF GitHub Bot Created on: 14/Jun/21 07:42 Start Date: 14/Jun/21 07:42 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3095: URL: https://github.com/apache/hadoop/pull/3095#issuecomment-859933114 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 36s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 34m 3s | | trunk passed | | +1 :green_heart: | compile | 1m 44s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | +1 :green_heart: | checkstyle | 1m 19s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 52s | | trunk passed | | +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | +1 :green_heart: | spotbugs | 4m 15s | | trunk passed | | +1 :green_heart: | shadedclient | 22m 17s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 32s | | the patch passed | | +1 :green_heart: | compile | 1m 39s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 1m 39s | | the patch passed | | +1 :green_heart: | compile | 1m 27s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | +1 :green_heart: | javac | 1m 27s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 1m 8s | | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 86 unchanged - 2 fixed = 86 total (was 88) | | +1 :green_heart: | mvnsite | 1m 38s | | the patch passed | | +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 47s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | +1 :green_heart: | spotbugs | 4m 30s | | the patch passed | | +1 :green_heart: | shadedclient | 22m 33s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | -1 :x: | unit | 251m 16s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3095/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 48s | | The patch does not generate ASF License warnings. | | | | 356m 0s | | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor | | | hadoop.hdfs.TestDFSStripedInputStream | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3095/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/3095 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux fa237f688cc1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / 4040ad64a4537d47e3633d23daaaefe7e9d7192a | | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | Test Results | https://ci-hadoop.apache.org/job/hadoo
[jira] [Work logged] (HDFS-16061) DFTestUtil.waitReplication can produce false positives
[ https://issues.apache.org/jira/browse/HDFS-16061?focusedWorklogId=609915&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609915 ] ASF GitHub Bot logged work on HDFS-16061: - Author: ASF GitHub Bot Created on: 10/Jun/21 20:24 Start Date: 10/Jun/21 20:24 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3095: URL: https://github.com/apache/hadoop/pull/3095#issuecomment-859025279 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 32s | | Docker mode activated. | _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 1s | | codespell was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. | _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 30m 41s | | trunk passed | | +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | +1 :green_heart: | checkstyle | 1m 2s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 24s | | trunk passed | | +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | +1 :green_heart: | spotbugs | 3m 9s | | trunk passed | | +1 :green_heart: | shadedclient | 16m 18s | | branch has no errors when building and testing our client artifacts. | _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 11s | | the patch passed | | +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 1m 11s | | the patch passed | | +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | +1 :green_heart: | javac | 1m 6s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 53s | | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 86 unchanged - 2 fixed = 86 total (was 88) | | +1 :green_heart: | mvnsite | 1m 11s | | the patch passed | | +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | +1 :green_heart: | spotbugs | 3m 8s | | the patch passed | | +1 :green_heart: | shadedclient | 16m 7s | | patch has no errors when building and testing our client artifacts. | _ Other Tests _ | | -1 :x: | unit | 252m 56s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3095/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 45s | | The patch does not generate ASF License warnings. | | | | 336m 55s | | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.TestFileCorruption | | | hadoop.hdfs.TestDecommissionWithBackoffMonitor | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3095/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/3095 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell | | uname | Linux a756d5c27042 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / da16c59f102f81ff2be92b85f10f487fb7b2541c | | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranc
[jira] [Work logged] (HDFS-16061) DFTestUtil.waitReplication can produce false positives
[ https://issues.apache.org/jira/browse/HDFS-16061?focusedWorklogId=609742&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609742 ] ASF GitHub Bot logged work on HDFS-16061: - Author: ASF GitHub Bot Created on: 10/Jun/21 14:46 Start Date: 10/Jun/21 14:46 Worklog Time Spent: 10m Work Description: amahussein opened a new pull request #3095: URL: https://github.com/apache/hadoop/pull/3095 HDFS-16061 DFTestUtil.waitReplication can produce false positives While checking the intermittent failure in `TestBalancerRPCDelay#testBalancerRPCDelayQpsDefault` described in HDFS-15146, I found that the implementation of waitReplication is incorrect. In the last iteration, when correctReplFactor is false, the thread sleeps for 1 second, then a TimeoutException is thrown without check whether the replication was complete in the last second. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 609742) Remaining Estimate: 0h Time Spent: 10m > DFTestUtil.waitReplication can produce false positives > -- > > Key: HDFS-16061 > URL: https://issues.apache.org/jira/browse/HDFS-16061 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > While checking the intermittent failure in > TestBalancerRPCDelay#testBalancerRPCDelayQpsDefault described in HDFS-15146, > I found that the implementation of waitReplication is incorrect. > In the last iteration, when {{correctReplFactor}} is {{false}}, the thread > sleeps for 1 second, then a {{TimeoutException}} is thrown without check > whether the replication was complete in the last second. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org