[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867099#comment-16867099
]
Nikhil Navadiya commented on HDFS-14101:
Thank you Wei-Chiu for the commit :)
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL: https://issues.apache.org/jira/browse/HDFS-14101
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.2.0, 3.0.3, 2.8.5
>Reporter: Kihwal Lee
>Assignee: Zsolt Venczel
>Priority: Major
> Labels: newbie
> Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3
>
> Attachments: HDFS-14101-branch-3.2.03.patch,
> HDFS-14101-branch-3.2.04.patch, HDFS-14101.01.patch, HDFS-14101.02.patch
>
>
> We've seen this occasionally.
> {noformat}
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755)
> at org.apache.hadoop.hdfs.server.namenode.
>
> TestListCorruptFileBlocks.testListCorruptFilesCorruptedBlock(TestListCorruptFileBlocks.java:105)
> {noformat}
> The test has a flaw.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866998#comment-16866998
]
Hadoop QA commented on HDFS-14101:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
28s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 6s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.server.namenode.TestEditLogRace |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be |
| JIRA Issue | HDFS-14101 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12972123/HDFS-14101-branch-3.2.04.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e047f846932b 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / 223a582 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26995/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26995/testReport/ |
| Max. process+thread count | 4530 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866886#comment-16866886
]
Wei-Chiu Chuang commented on HDFS-14101:
Thanks [~nikhil.navadiya]. +1 on v04 pending Jenkins.
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL: https://issues.apache.org/jira/browse/HDFS-14101
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.2.0, 3.0.3, 2.8.5
>Reporter: Kihwal Lee
>Assignee: Zsolt Venczel
>Priority: Major
> Labels: newbie
> Fix For: 3.3.0
>
> Attachments: HDFS-14101-branch-3.2.03.patch,
> HDFS-14101-branch-3.2.04.patch, HDFS-14101.01.patch, HDFS-14101.02.patch
>
>
> We've seen this occasionally.
> {noformat}
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755)
> at org.apache.hadoop.hdfs.server.namenode.
>
> TestListCorruptFileBlocks.testListCorruptFilesCorruptedBlock(TestListCorruptFileBlocks.java:105)
> {noformat}
> The test has a flaw.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866857#comment-16866857
]
Nikhil Navadiya commented on HDFS-14101:
Hi All,
I verified that following tests are running fine on my local machine
hadoop.hdfs.web.TestWebHdfsTimeouts,
hadoop.hdfs.server.namenode.ha.TestEditLogTailer,
hadoop.hdfs.TestReconstructStripedFile,
hadoop.hdfs.server.namenode.sps.TestBlockStorageMovementAttemptedItems
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL: https://issues.apache.org/jira/browse/HDFS-14101
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.2.0, 3.0.3, 2.8.5
>Reporter: Kihwal Lee
>Assignee: Zsolt Venczel
>Priority: Major
> Labels: newbie
> Fix For: 3.3.0
>
> Attachments: HDFS-14101-branch-3.2.03.patch, HDFS-14101.01.patch,
> HDFS-14101.02.patch
>
>
> We've seen this occasionally.
> {noformat}
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755)
> at org.apache.hadoop.hdfs.server.namenode.
>
> TestListCorruptFileBlocks.testListCorruptFilesCorruptedBlock(TestListCorruptFileBlocks.java:105)
> {noformat}
> The test has a flaw.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866143#comment-16866143
]
Hadoop QA commented on HDFS-14101:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
49s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}122m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
56s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be |
| JIRA Issue | HDFS-14101 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12972017/HDFS-14101-branch-3.2.03.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6af68cf3a5ac 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / 0eb42e1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26978/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26978/testReport/ |
| Max. process+thread count | 3961 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720142#comment-16720142
]
Zsolt Venczel commented on HDFS-14101:
--
Thanks a lot [~mackrorysd] for the meaningful note and the commit!
We might want to increase the corruption length or not use random at all in
unit tests?
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL: https://issues.apache.org/jira/browse/HDFS-14101
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.2.0, 3.0.3, 2.8.5
>Reporter: Kihwal Lee
>Assignee: Zsolt Venczel
>Priority: Major
> Labels: newbie
> Fix For: 3.3.0
>
> Attachments: HDFS-14101.01.patch, HDFS-14101.02.patch
>
>
> We've seen this occasionally.
> {noformat}
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755)
> at org.apache.hadoop.hdfs.server.namenode.
>
> TestListCorruptFileBlocks.testListCorruptFilesCorruptedBlock(TestListCorruptFileBlocks.java:105)
> {noformat}
> The test has a flaw.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715584#comment-16715584
]
Hudson commented on HDFS-14101:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15583 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15583/])
HDFS-14101. Fixing underflow error in test. Contributed by Zsolt (mackrorysd:
rev 80e59e7876fed1c9d56e696331e0c54e7cd3499b)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL: https://issues.apache.org/jira/browse/HDFS-14101
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.2.0, 3.0.3, 2.8.5
>Reporter: Kihwal Lee
>Assignee: Zsolt Venczel
>Priority: Major
> Labels: newbie
> Fix For: 3.3.0
>
> Attachments: HDFS-14101.01.patch, HDFS-14101.02.patch
>
>
> We've seen this occasionally.
> {noformat}
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755)
> at org.apache.hadoop.hdfs.server.namenode.
>
> TestListCorruptFileBlocks.testListCorruptFilesCorruptedBlock(TestListCorruptFileBlocks.java:105)
> {noformat}
> The test has a flaw.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714982#comment-16714982
]
Sean Mackrory commented on HDFS-14101:
--
+1, thanks Zsolt. I think the shared variable is an improvement, but I still
think it's worth a comment, so I'm gonna add the following add the declaration
of corruptionLength if no one objects:
{code}
// Files are corrupted with 2 bytes before the end of the file,
// so that's the minimum length
{code}
Of course, as I understand it, there's still a 1:65,536 chance the corruption
isn't detected because the random bytes we overwrite with are identical to
those that were there originally, but this is a step in the right direction :)
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL: https://issues.apache.org/jira/browse/HDFS-14101
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.2.0, 3.0.3, 2.8.5
>Reporter: Kihwal Lee
>Assignee: Zsolt Venczel
>Priority: Major
> Labels: newbie
> Attachments: HDFS-14101.01.patch, HDFS-14101.02.patch
>
>
> We've seen this occasionally.
> {noformat}
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755)
> at org.apache.hadoop.hdfs.server.namenode.
>
> TestListCorruptFileBlocks.testListCorruptFilesCorruptedBlock(TestListCorruptFileBlocks.java:105)
> {noformat}
> The test has a flaw.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703182#comment-16703182
]
Hadoop QA commented on HDFS-14101:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 6s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.namenode.sps.TestBlockStorageMovementAttemptedItems |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14101 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949997/HDFS-14101.02.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 697785dd1cfd 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b71cc7f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25668/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25668/testReport/ |
| Max. process+thread count | 3939 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703048#comment-16703048
]
Zsolt Venczel commented on HDFS-14101:
--
Thanks for the review [~ayushtkn] and for the valuable comment.
In the latest patch I was trying to make the relation between the minimum file
size and the size of the data meant to corrupt the block more clear by shared
constants.
I hope this way it's more meaningful than a comment.
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL: https://issues.apache.org/jira/browse/HDFS-14101
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.2.0, 3.0.3, 2.8.5
>Reporter: Kihwal Lee
>Assignee: Zsolt Venczel
>Priority: Major
> Labels: newbie
> Attachments: HDFS-14101.01.patch, HDFS-14101.02.patch
>
>
> We've seen this occasionally.
> {noformat}
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755)
> at org.apache.hadoop.hdfs.server.namenode.
>
> TestListCorruptFileBlocks.testListCorruptFilesCorruptedBlock(TestListCorruptFileBlocks.java:105)
> {noformat}
> The test has a flaw.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700853#comment-16700853
]
Ayush Saxena commented on HDFS-14101:
-
Thanks [~kihwal] and [~zvenczel] for the analysis.
I too think that is only the case causing it a failure.The probability of
selecting 1 from total 512 values is quite less.I guess that is why we didn't
see it quite frequently.
Would you mind adding even a small comment there too so as to mention the
reason for choosing two not one.Might be little helpful for some one going
through it in the future. :)
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL: https://issues.apache.org/jira/browse/HDFS-14101
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.2.0, 3.0.3, 2.8.5
>Reporter: Kihwal Lee
>Assignee: Zsolt Venczel
>Priority: Major
> Labels: newbie
> Attachments: HDFS-14101.01.patch
>
>
> We've seen this occasionally.
> {noformat}
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755)
> at org.apache.hadoop.hdfs.server.namenode.
>
> TestListCorruptFileBlocks.testListCorruptFilesCorruptedBlock(TestListCorruptFileBlocks.java:105)
> {noformat}
> The test has a flaw.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700559#comment-16700559
]
Hadoop QA commented on HDFS-14101:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 78m
47s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14101 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949677/HDFS-14101.01.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 736818d3184a 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 96c104d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25640/testReport/ |
| Max. process+thread count | 3099 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25640/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL:
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700374#comment-16700374
]
Zsolt Venczel commented on HDFS-14101:
--
Thanks [~kihwal] for reporting the issue!
DFSTestUtil.Builder creates files with random size no larger then 512 bytes and
no smaller then 1 byte.
{code}
DFSTestUtil util = new DFSTestUtil.Builder().
setName("testCorruptFilesCorruptedBlock").setNumFiles(2).
setMaxLevels(1).setMaxSize(512).build();
{code}
Whenever the file size is 1 byte the test fails as it tries to corrupt a block
by inserting a 2 bytes long buffer starting 2 bytes before the end of the file
that is -1.
The submitted patch should fix this (statistically this test had failed 1 times
per 512 run but it was running fine for 2000 runs having the patch).
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL: https://issues.apache.org/jira/browse/HDFS-14101
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.2.0, 3.0.3, 2.8.5
>Reporter: Kihwal Lee
>Assignee: Zsolt Venczel
>Priority: Major
> Labels: newbie
> Attachments: HDFS-14101.01.patch
>
>
> We've seen this occasionally.
> {noformat}
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755)
> at org.apache.hadoop.hdfs.server.namenode.
>
> TestListCorruptFileBlocks.testListCorruptFilesCorruptedBlock(TestListCorruptFileBlocks.java:105)
> {noformat}
> The test has a flaw.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699563#comment-16699563
]
Kihwal Lee commented on HDFS-14101:
---
The test is very old, pre-2011, pre-git. The source of this bug may be traced
using the old svn repo, but I won't go there.
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL: https://issues.apache.org/jira/browse/HDFS-14101
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.5
>Reporter: Kihwal Lee
>Priority: Major
>
> We've seen this occasionally.
> {noformat}
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755)
> at org.apache.hadoop.hdfs.server.namenode.
>
> TestListCorruptFileBlocks.testListCorruptFilesCorruptedBlock(TestListCorruptFileBlocks.java:105)
> {noformat}
> The test has a flaw.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14101) Random failure of testListCorruptFilesCorruptedBlock
[
https://issues.apache.org/jira/browse/HDFS-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699560#comment-16699560
]
Kihwal Lee commented on HDFS-14101:
---
The test creates two files with random sizes. In a failed run, one was 295
bytes and the other 1 byte.
Then the meta file is overwritten to corrupt the block.
{code:java}
List metaFiles = MiniDFSCluster.getAllBlockFiles(data_dir);
...
File metaFile = metaFiles.get(0);
RandomAccessFile file = new RandomAccessFile(metaFile, "rw");
FileChannel channel = file.getChannel();
long position = channel.size() - 2;
...
channel.write(ByteBuffer.wrap(buffer), position);
{code}
At a glance, the meta file cannot be 0 or 1 byte long, since it has a header.
But the implementation of {{getAllBlockFiles()}} does not return meta files. It
actually returns list of data files.
{code:java}
for (File f : files) {
if (f.getName().startsWith(Block.BLOCK_FILE_PREFIX) &&
!f.getName().endsWith(Block.METADATA_EXTENSION)) {
results.add(f);
...
}
{code}
So this test fails if the random test file length happens to be too small.
> Random failure of testListCorruptFilesCorruptedBlock
>
>
> Key: HDFS-14101
> URL: https://issues.apache.org/jira/browse/HDFS-14101
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.5
>Reporter: Kihwal Lee
>Priority: Major
>
> We've seen this occasionally.
> {noformat}
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755)
> at org.apache.hadoop.hdfs.server.namenode.
>
> TestListCorruptFileBlocks.testListCorruptFilesCorruptedBlock(TestListCorruptFileBlocks.java:105)
> {noformat}
> The test has a flaw.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
