[jira] [Commented] (HDFS-11964) RS-6-3-LEGACY has a decoding bug when it is used for pread
[ https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149491#comment-16149491 ] Hadoop QA commented on HDFS-11964: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 13s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}126m 31s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 27s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}163m 18s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | | | hadoop.hdfs.TestDFSUpgrade | | | hadoop.hdfs.TestLeaseRecoveryStriped | | | hadoop.hdfs.server.mover.TestMover | | | hadoop.hdfs.TestClientProtocolForPipelineRecovery | | | hadoop.hdfs.security.token.block.TestBlockToken | | | hadoop.hdfs.TestReconstructStripedFile | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 | | | hadoop.hdfs.TestReadStripedFileWithMissingBlocks | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 | | | hadoop.hdfs.qjournal.server.TestJournalNodeSync | | | hadoop.hdfs.TestParallelRead | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 | | |
[jira] [Commented] (HDFS-11964) RS-6-3-LEGACY has a decoding bug when it is used for pread
[ https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146708#comment-16146708 ] Kai Zheng commented on HDFS-11964: -- Hi [~tasanuma0829], Thanks for your update digging into the deep. I thought you caught the root cause. One comment, in the below block the re-initialization of the {{codingBuffer}} isn't necessary because it has been done in {{initDecodeInputs}} already. +1 on addressing this. {code} +int bufLen = (int) alignedStripe.getSpanInBlock(); +int bufCount = dataBlkNum + parityBlkNum; +codingBuffer = dfsStripedInputStream.getBufferPool(). +getBuffer(useDirectBuffer(), bufLen * bufCount); +ByteBuffer buffer = codingBuffer.duplicate(); +decodeInputs[index] = new ECChunk(buffer, index * bufLen, bufLen); {code} > RS-6-3-LEGACY has a decoding bug when it is used for pread > -- > > Key: HDFS-11964 > URL: https://issues.apache.org/jira/browse/HDFS-11964 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Takanobu Asanuma > Attachments: HDFS-11964.1.patch, HDFS-11964.2.patch > > > TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure fails on > trunk: > {code} > Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.99 sec <<< > FAILURE! - in > org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > testPreadWithDNFailure(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy) > Time elapsed: 1.265 sec <<< FAILURE! > org.junit.internal.ArrayComparisonFailure: arrays first differed at element > [327680]; expected:<-36> but was:<2> > at > org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50) > at org.junit.Assert.internalArrayEquals(Assert.java:473) > at org.junit.Assert.assertArrayEquals(Assert.java:294) > at org.junit.Assert.assertArrayEquals(Assert.java:305) > at > org.apache.hadoop.hdfs.TestDFSStripedInputStream.testPreadWithDNFailure(TestDFSStripedInputStream.java:306) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11964) RS-6-3-LEGACY has a decoding bug when it is used for pread
[ https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145217#comment-16145217 ] Hadoop QA commented on HDFS-11964: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 11s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 5s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11964 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884240/HDFS-11964.2.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 9c7336f85da5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 33afa94 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20909/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20909/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > RS-6-3-LEGACY has a decoding bug when it is used for pread > -- > > Key: HDFS-11964 > URL: https://issues.apache.org/jira/browse/HDFS-11964 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Takanobu Asanuma > Attachments: HDFS-11964.1.patch, HDFS-11964.2.patch > > >
[jira] [Commented] (HDFS-11964) RS-6-3-LEGACY has a decoding bug when it is used for pread
[ https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143535#comment-16143535 ] Takanobu Asanuma commented on HDFS-11964: - Thanks for reviewing and the comment, [~drankye]! > We could also compare it with the default RS coder. {{RSRawDecoder#doDecode}} (the default RS coder) also receives unnecessary input arrays when it preads, but it does like a post fix, and decode blocks well. > I wonder if we could get right at the first place. I agree with you. Unlike {{StatefulStripeReader#prepareDecodeInputs()}}, {{PositionStripeReader#prepareDecodeInputs()}} seems not to generate necessary and sufficient input arrays. I will investigate it more. Thanks! > RS-6-3-LEGACY has a decoding bug when it is used for pread > -- > > Key: HDFS-11964 > URL: https://issues.apache.org/jira/browse/HDFS-11964 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Takanobu Asanuma > Attachments: HDFS-11964.1.patch > > > TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure fails on > trunk: > {code} > Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.99 sec <<< > FAILURE! - in > org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > testPreadWithDNFailure(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy) > Time elapsed: 1.265 sec <<< FAILURE! > org.junit.internal.ArrayComparisonFailure: arrays first differed at element > [327680]; expected:<-36> but was:<2> > at > org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50) > at org.junit.Assert.internalArrayEquals(Assert.java:473) > at org.junit.Assert.assertArrayEquals(Assert.java:294) > at org.junit.Assert.assertArrayEquals(Assert.java:305) > at > org.apache.hadoop.hdfs.TestDFSStripedInputStream.testPreadWithDNFailure(TestDFSStripedInputStream.java:306) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11964) RS-6-3-LEGACY has a decoding bug when it is used for pread
[ https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143250#comment-16143250 ] Kai Zheng commented on HDFS-11964: -- Hi [~tasanuma0829], Thanks for working on this. I understand the issue and your fix. It looks to me {{int[] updateErasedIndexes(int[] oldIndexes, int parityNum)}} is a post fix to the indexes array, I wonder if we could get right at the first place. The indexes array is out by the following call, so the root cause could be the inputs array isn't well set or initialized before doing the decode call, some of the entries should be set null to indicate the positions are erased/corrupt or just not to read. {code} int[] erasedOrNotToReadIndexes = CoderUtil.getNullIndexes(decodingState.inputs); {code} Could you investigate more in your debugging session? We could also compare it with the default RS coder. Thanks! > RS-6-3-LEGACY has a decoding bug when it is used for pread > -- > > Key: HDFS-11964 > URL: https://issues.apache.org/jira/browse/HDFS-11964 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Takanobu Asanuma > Attachments: HDFS-11964.1.patch > > > TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure fails on > trunk: > {code} > Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.99 sec <<< > FAILURE! - in > org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > testPreadWithDNFailure(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy) > Time elapsed: 1.265 sec <<< FAILURE! > org.junit.internal.ArrayComparisonFailure: arrays first differed at element > [327680]; expected:<-36> but was:<2> > at > org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50) > at org.junit.Assert.internalArrayEquals(Assert.java:473) > at org.junit.Assert.assertArrayEquals(Assert.java:294) > at org.junit.Assert.assertArrayEquals(Assert.java:305) > at > org.apache.hadoop.hdfs.TestDFSStripedInputStream.testPreadWithDNFailure(TestDFSStripedInputStream.java:306) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11964) RS-6-3-LEGACY has a decoding bug when it is used for pread
[ https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141031#comment-16141031 ] Kai Zheng commented on HDFS-11964: -- In annual leave and vacation, email response will be delayed. > RS-6-3-LEGACY has a decoding bug when it is used for pread > -- > > Key: HDFS-11964 > URL: https://issues.apache.org/jira/browse/HDFS-11964 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Takanobu Asanuma > Attachments: HDFS-11964.1.patch > > > TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure fails on > trunk: > {code} > Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.99 sec <<< > FAILURE! - in > org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > testPreadWithDNFailure(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy) > Time elapsed: 1.265 sec <<< FAILURE! > org.junit.internal.ArrayComparisonFailure: arrays first differed at element > [327680]; expected:<-36> but was:<2> > at > org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50) > at org.junit.Assert.internalArrayEquals(Assert.java:473) > at org.junit.Assert.assertArrayEquals(Assert.java:294) > at org.junit.Assert.assertArrayEquals(Assert.java:305) > at > org.apache.hadoop.hdfs.TestDFSStripedInputStream.testPreadWithDNFailure(TestDFSStripedInputStream.java:306) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11964) RS-6-3-LEGACY has a decoding bug when it is used for pread
[ https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141029#comment-16141029 ] Takanobu Asanuma commented on HDFS-11964: - Thanks for the comment, [~drankye]. No, I'm not using the legacy coder. > RS-6-3-LEGACY has a decoding bug when it is used for pread > -- > > Key: HDFS-11964 > URL: https://issues.apache.org/jira/browse/HDFS-11964 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Takanobu Asanuma > Attachments: HDFS-11964.1.patch > > > TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure fails on > trunk: > {code} > Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.99 sec <<< > FAILURE! - in > org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > testPreadWithDNFailure(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy) > Time elapsed: 1.265 sec <<< FAILURE! > org.junit.internal.ArrayComparisonFailure: arrays first differed at element > [327680]; expected:<-36> but was:<2> > at > org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50) > at org.junit.Assert.internalArrayEquals(Assert.java:473) > at org.junit.Assert.assertArrayEquals(Assert.java:294) > at org.junit.Assert.assertArrayEquals(Assert.java:305) > at > org.apache.hadoop.hdfs.TestDFSStripedInputStream.testPreadWithDNFailure(TestDFSStripedInputStream.java:306) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11964) RS-6-3-LEGACY has a decoding bug when it is used for pread
[ https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140915#comment-16140915 ] Kai Zheng commented on HDFS-11964: -- Hi [~tasanuma0829], Thanks for the ping, yes I will give it a review soon. A quick question, are you using the legacy coder? Note the using of it is discouraged if there's not legacy considerations. > RS-6-3-LEGACY has a decoding bug when it is used for pread > -- > > Key: HDFS-11964 > URL: https://issues.apache.org/jira/browse/HDFS-11964 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Takanobu Asanuma > Attachments: HDFS-11964.1.patch > > > TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure fails on > trunk: > {code} > Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.99 sec <<< > FAILURE! - in > org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > testPreadWithDNFailure(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy) > Time elapsed: 1.265 sec <<< FAILURE! > org.junit.internal.ArrayComparisonFailure: arrays first differed at element > [327680]; expected:<-36> but was:<2> > at > org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50) > at org.junit.Assert.internalArrayEquals(Assert.java:473) > at org.junit.Assert.assertArrayEquals(Assert.java:294) > at org.junit.Assert.assertArrayEquals(Assert.java:305) > at > org.apache.hadoop.hdfs.TestDFSStripedInputStream.testPreadWithDNFailure(TestDFSStripedInputStream.java:306) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11964) RS-6-3-LEGACY has a decoding bug when it is used for pread
[ https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139483#comment-16139483 ] Hadoop QA commented on HDFS-11964: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 29s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 33s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 69m 53s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestKDiag | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11964 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12883464/HDFS-11964.1.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 010812fd7bd3 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 652dd43 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20832/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20832/testReport/ | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20832/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > RS-6-3-LEGACY has a decoding bug when it is used for pread > -- > > Key: HDFS-11964 > URL: https://issues.apache.org/jira/browse/HDFS-11964 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions:
[jira] [Commented] (HDFS-11964) RS-6-3-LEGACY has a decoding bug when it is used for pread
[ https://issues.apache.org/jira/browse/HDFS-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139418#comment-16139418 ] Takanobu Asanuma commented on HDFS-11964: - I found the cause of the bug. If clients do pread an ec file whose block group have some missing/corrupted blocks, {{RSLegacyRawDecoder#doDecode}} decodes blocks with not enough {{erasedOrNotToReadIndexes}}. In this case, we need to update {{erasedOrNotToReadIndexes}}. (About this failed test case, we need to update \{8\} with \{1, 2, 8\}.) > RS-6-3-LEGACY has a decoding bug when it is used for pread > -- > > Key: HDFS-11964 > URL: https://issues.apache.org/jira/browse/HDFS-11964 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Takanobu Asanuma > > TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure fails on > trunk: > {code} > Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.99 sec <<< > FAILURE! - in > org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy > testPreadWithDNFailure(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy) > Time elapsed: 1.265 sec <<< FAILURE! > org.junit.internal.ArrayComparisonFailure: arrays first differed at element > [327680]; expected:<-36> but was:<2> > at > org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50) > at org.junit.Assert.internalArrayEquals(Assert.java:473) > at org.junit.Assert.assertArrayEquals(Assert.java:294) > at org.junit.Assert.assertArrayEquals(Assert.java:305) > at > org.apache.hadoop.hdfs.TestDFSStripedInputStream.testPreadWithDNFailure(TestDFSStripedInputStream.java:306) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org