[ https://issues.apache.org/jira/browse/HDFS-17547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ruiliang updated HDFS-17547: ---------------------------- Attachment: image-2024-06-07-16-02-07-480.png Description: When I validate a block that has been corrupted many times, does it appear normal? {code:java} hdfs debug verifyEC -file /file.orc 24/06/07 15:40:29 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable Checking EC block group: blk_-9223372036492703744 Status: OK {code} ByteBuffer hb show [0......] [0......] !image-2024-06-07-16-02-07-480.png! {code:java} buffers = {ByteBuffer[5]@3270} 0 = {HeapByteBuffer@3430} "java.nio.HeapByteBuffer[pos=65536 lim=65536 cap=65536]" 1 = {HeapByteBuffer@3434} "java.nio.HeapByteBuffer[pos=65536 lim=65536 cap=65536]" 2 = {HeapByteBuffer@3438} "java.nio.HeapByteBuffer[pos=65536 lim=65536 cap=65536]" 3 = {HeapByteBuffer@3504} "java.nio.HeapByteBuffer[pos=0 lim=65536 cap=65536]" hb = {byte[65536]@3511} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +65,436 more] buffers[this.dataBlkNum + ixx].equals(outputs[ixx] =true ? outputs = {ByteBuffer[2]@3271} 0 = {HeapByteBuffer@3455} "java.nio.HeapByteBuffer[pos=0 lim=65536 cap=65536]" hb = {byte[65536]@3459} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +65,436 more]{code} Can this situation be judged as an anomaly? check orc file {code:java} Structure for skip_ip/_skip_file File Version: 0.12 with ORC_517 by ORC Java Exception in thread "main" java.io.IOException: Problem opening stripe 0 footer in skip_ip/_skip_file. at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:360) at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:879) at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:873) at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:345) at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:276) at org.apache.orc.tools.FileDump.main(FileDump.java:137) at org.apache.orc.tools.Driver.main(Driver.java:124) Caused by: java.lang.IllegalArgumentException: Buffer size too small. size = 131072 needed = 7752508 in column 3 kind LENGTH at org.apache.orc.impl.InStream$CompressedStream.readHeader(InStream.java:481) at org.apache.orc.impl.InStream$CompressedStream.ensureUncompressed(InStream.java:528) at org.apache.orc.impl.InStream$CompressedStream.read(InStream.java:507) at org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:59) at org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:333) at org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.readDictionaryLengthStream(TreeReaderFactory.java:2221) at org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.startStripe(TreeReaderFactory.java:2201) at org.apache.orc.impl.TreeReaderFactory$StringTreeReader.startStripe(TreeReaderFactory.java:1943) at org.apache.orc.impl.reader.tree.StructBatchReader.startStripe(StructBatchReader.java:112) at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1251) at org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1290) at org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1333) at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:355) ... 6 more {code} was: When I validate a block that has been corrupted many times, does it appear normal? {code:java} hdfs debug verifyEC -file /file.orc 24/06/07 15:40:29 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable Checking EC block group: blk_-9223372036492703744 Status: OK {code} ByteBuffer hb show [0......] {code:java} buffers = {ByteBuffer[5]@3270} 0 = {HeapByteBuffer@3430} "java.nio.HeapByteBuffer[pos=65536 lim=65536 cap=65536]" 1 = {HeapByteBuffer@3434} "java.nio.HeapByteBuffer[pos=65536 lim=65536 cap=65536]" 2 = {HeapByteBuffer@3438} "java.nio.HeapByteBuffer[pos=65536 lim=65536 cap=65536]" 3 = {HeapByteBuffer@3504} "java.nio.HeapByteBuffer[pos=0 lim=65536 cap=65536]" hb = {byte[65536]@3511} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +65,436 more] buffers[this.dataBlkNum + ixx].equals(outputs[ixx] =true ? outputs = {ByteBuffer[2]@3271} 0 = {HeapByteBuffer@3455} "java.nio.HeapByteBuffer[pos=0 lim=65536 cap=65536]" hb = {byte[65536]@3459} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, +65,436 more]{code} Can this situation be judged as an anomaly? check orc file {code:java} Structure for skip_ip/_skip_file File Version: 0.12 with ORC_517 by ORC Java Exception in thread "main" java.io.IOException: Problem opening stripe 0 footer in skip_ip/_skip_file. at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:360) at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:879) at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:873) at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:345) at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:276) at org.apache.orc.tools.FileDump.main(FileDump.java:137) at org.apache.orc.tools.Driver.main(Driver.java:124) Caused by: java.lang.IllegalArgumentException: Buffer size too small. size = 131072 needed = 7752508 in column 3 kind LENGTH at org.apache.orc.impl.InStream$CompressedStream.readHeader(InStream.java:481) at org.apache.orc.impl.InStream$CompressedStream.ensureUncompressed(InStream.java:528) at org.apache.orc.impl.InStream$CompressedStream.read(InStream.java:507) at org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:59) at org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:333) at org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.readDictionaryLengthStream(TreeReaderFactory.java:2221) at org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.startStripe(TreeReaderFactory.java:2201) at org.apache.orc.impl.TreeReaderFactory$StringTreeReader.startStripe(TreeReaderFactory.java:1943) at org.apache.orc.impl.reader.tree.StructBatchReader.startStripe(StructBatchReader.java:112) at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1251) at org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1290) at org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1333) at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:355) ... 6 more {code} > debug verifyEC check error > -------------------------- > > Key: HDFS-17547 > URL: https://issues.apache.org/jira/browse/HDFS-17547 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-common > Reporter: ruiliang > Priority: Minor > Attachments: image-2024-06-07-16-02-07-480.png > > > When I validate a block that has been corrupted many times, does it appear > normal? > > {code:java} > hdfs debug verifyEC -file /file.orc > 24/06/07 15:40:29 WARN erasurecode.ErasureCodeNative: ISA-L support is not > available in your platform... using builtin-java codec where applicable > Checking EC block group: blk_-9223372036492703744 > Status: OK > {code} > > > ByteBuffer hb show [0......] [0......] > !image-2024-06-07-16-02-07-480.png! > {code:java} > buffers = {ByteBuffer[5]@3270} > 0 = {HeapByteBuffer@3430} "java.nio.HeapByteBuffer[pos=65536 lim=65536 > cap=65536]" > 1 = {HeapByteBuffer@3434} "java.nio.HeapByteBuffer[pos=65536 lim=65536 > cap=65536]" > 2 = {HeapByteBuffer@3438} "java.nio.HeapByteBuffer[pos=65536 lim=65536 > cap=65536]" > 3 = {HeapByteBuffer@3504} "java.nio.HeapByteBuffer[pos=0 lim=65536 > cap=65536]" > hb = {byte[65536]@3511} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, +65,436 more] > buffers[this.dataBlkNum + ixx].equals(outputs[ixx] =true ? > outputs = {ByteBuffer[2]@3271} > 0 = {HeapByteBuffer@3455} "java.nio.HeapByteBuffer[pos=0 lim=65536 > cap=65536]" > hb = {byte[65536]@3459} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, +65,436 more]{code} > Can this situation be judged as an anomaly? > > check orc file > {code:java} > Structure for skip_ip/_skip_file File Version: 0.12 with ORC_517 by ORC Java > Exception in thread "main" java.io.IOException: Problem opening stripe 0 > footer in skip_ip/_skip_file. at > org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:360) > at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:879) at > org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:873) at > org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:345) at > org.apache.orc.tools.FileDump.printMetaData(FileDump.java:276) at > org.apache.orc.tools.FileDump.main(FileDump.java:137) at > org.apache.orc.tools.Driver.main(Driver.java:124) Caused by: > java.lang.IllegalArgumentException: Buffer size too small. size = 131072 > needed = 7752508 in column 3 kind LENGTH at > org.apache.orc.impl.InStream$CompressedStream.readHeader(InStream.java:481) > at > org.apache.orc.impl.InStream$CompressedStream.ensureUncompressed(InStream.java:528) > at > org.apache.orc.impl.InStream$CompressedStream.read(InStream.java:507) > at > org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:59) > at > org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:333) > at > org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.readDictionaryLengthStream(TreeReaderFactory.java:2221) > at > org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.startStripe(TreeReaderFactory.java:2201) > at > org.apache.orc.impl.TreeReaderFactory$StringTreeReader.startStripe(TreeReaderFactory.java:1943) > at > org.apache.orc.impl.reader.tree.StructBatchReader.startStripe(StructBatchReader.java:112) > at > org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1251) > at > org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1290) > at > org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1333) > at > org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:355) > ... 6 more > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org