[
https://issues.apache.org/jira/browse/HDFS-17547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ruiliang updated HDFS-17547:
----------------------------
Attachment: image-2024-06-07-16-02-07-480.png
Description:
When I validate a block that has been corrupted many times, does it appear
normal?
{code:java}
hdfs debug verifyEC -file /file.orc
24/06/07 15:40:29 WARN erasurecode.ErasureCodeNative: ISA-L support is not
available in your platform... using builtin-java codec where applicable
Checking EC block group: blk_-9223372036492703744
Status: OK
{code}
ByteBuffer hb show [0......] [0......]
!image-2024-06-07-16-02-07-480.png!
{code:java}
buffers = {ByteBuffer[5]@3270}
0 = {HeapByteBuffer@3430} "java.nio.HeapByteBuffer[pos=65536 lim=65536
cap=65536]"
1 = {HeapByteBuffer@3434} "java.nio.HeapByteBuffer[pos=65536 lim=65536
cap=65536]"
2 = {HeapByteBuffer@3438} "java.nio.HeapByteBuffer[pos=65536 lim=65536
cap=65536]"
3 = {HeapByteBuffer@3504} "java.nio.HeapByteBuffer[pos=0 lim=65536 cap=65536]"
hb = {byte[65536]@3511} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, +65,436 more]
buffers[this.dataBlkNum + ixx].equals(outputs[ixx] =true ?
outputs = {ByteBuffer[2]@3271}
0 = {HeapByteBuffer@3455} "java.nio.HeapByteBuffer[pos=0 lim=65536 cap=65536]"
hb = {byte[65536]@3459} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, +65,436 more]{code}
Can this situation be judged as an anomaly?
check orc file
{code:java}
Structure for skip_ip/_skip_file File Version: 0.12 with ORC_517 by ORC Java
Exception in thread "main" java.io.IOException: Problem opening stripe 0 footer
in skip_ip/_skip_file. at
org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:360)
at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:879) at
org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:873) at
org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:345) at
org.apache.orc.tools.FileDump.printMetaData(FileDump.java:276) at
org.apache.orc.tools.FileDump.main(FileDump.java:137) at
org.apache.orc.tools.Driver.main(Driver.java:124) Caused by:
java.lang.IllegalArgumentException: Buffer size too small. size = 131072 needed
= 7752508 in column 3 kind LENGTH at
org.apache.orc.impl.InStream$CompressedStream.readHeader(InStream.java:481)
at
org.apache.orc.impl.InStream$CompressedStream.ensureUncompressed(InStream.java:528)
at
org.apache.orc.impl.InStream$CompressedStream.read(InStream.java:507)
at
org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:59)
at
org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:333)
at
org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.readDictionaryLengthStream(TreeReaderFactory.java:2221)
at
org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.startStripe(TreeReaderFactory.java:2201)
at
org.apache.orc.impl.TreeReaderFactory$StringTreeReader.startStripe(TreeReaderFactory.java:1943)
at
org.apache.orc.impl.reader.tree.StructBatchReader.startStripe(StructBatchReader.java:112)
at
org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1251)
at
org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1290)
at
org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1333)
at
org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:355)
... 6 more
{code}
was:
When I validate a block that has been corrupted many times, does it appear
normal?
{code:java}
hdfs debug verifyEC -file /file.orc
24/06/07 15:40:29 WARN erasurecode.ErasureCodeNative: ISA-L support is not
available in your platform... using builtin-java codec where applicable
Checking EC block group: blk_-9223372036492703744
Status: OK
{code}
ByteBuffer hb show [0......]
{code:java}
buffers = {ByteBuffer[5]@3270}
0 = {HeapByteBuffer@3430} "java.nio.HeapByteBuffer[pos=65536 lim=65536
cap=65536]"
1 = {HeapByteBuffer@3434} "java.nio.HeapByteBuffer[pos=65536 lim=65536
cap=65536]"
2 = {HeapByteBuffer@3438} "java.nio.HeapByteBuffer[pos=65536 lim=65536
cap=65536]"
3 = {HeapByteBuffer@3504} "java.nio.HeapByteBuffer[pos=0 lim=65536 cap=65536]"
hb = {byte[65536]@3511} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, +65,436 more]
buffers[this.dataBlkNum + ixx].equals(outputs[ixx] =true ?
outputs = {ByteBuffer[2]@3271}
0 = {HeapByteBuffer@3455} "java.nio.HeapByteBuffer[pos=0 lim=65536 cap=65536]"
hb = {byte[65536]@3459} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, +65,436 more]{code}
Can this situation be judged as an anomaly?
check orc file
{code:java}
Structure for skip_ip/_skip_file File Version: 0.12 with ORC_517 by ORC Java
Exception in thread "main" java.io.IOException: Problem opening stripe 0 footer
in skip_ip/_skip_file. at
org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:360)
at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:879) at
org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:873) at
org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:345) at
org.apache.orc.tools.FileDump.printMetaData(FileDump.java:276) at
org.apache.orc.tools.FileDump.main(FileDump.java:137) at
org.apache.orc.tools.Driver.main(Driver.java:124) Caused by:
java.lang.IllegalArgumentException: Buffer size too small. size = 131072 needed
= 7752508 in column 3 kind LENGTH at
org.apache.orc.impl.InStream$CompressedStream.readHeader(InStream.java:481)
at
org.apache.orc.impl.InStream$CompressedStream.ensureUncompressed(InStream.java:528)
at
org.apache.orc.impl.InStream$CompressedStream.read(InStream.java:507)
at
org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:59)
at
org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:333)
at
org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.readDictionaryLengthStream(TreeReaderFactory.java:2221)
at
org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.startStripe(TreeReaderFactory.java:2201)
at
org.apache.orc.impl.TreeReaderFactory$StringTreeReader.startStripe(TreeReaderFactory.java:1943)
at
org.apache.orc.impl.reader.tree.StructBatchReader.startStripe(StructBatchReader.java:112)
at
org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1251)
at
org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1290)
at
org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1333)
at
org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:355)
... 6 more
{code}
> debug verifyEC check error
> --------------------------
>
> Key: HDFS-17547
> URL: https://issues.apache.org/jira/browse/HDFS-17547
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-common
> Reporter: ruiliang
> Priority: Minor
> Attachments: image-2024-06-07-16-02-07-480.png
>
>
> When I validate a block that has been corrupted many times, does it appear
> normal?
>
> {code:java}
> hdfs debug verifyEC -file /file.orc
> 24/06/07 15:40:29 WARN erasurecode.ErasureCodeNative: ISA-L support is not
> available in your platform... using builtin-java codec where applicable
> Checking EC block group: blk_-9223372036492703744
> Status: OK
> {code}
>
>
> ByteBuffer hb show [0......] [0......]
> !image-2024-06-07-16-02-07-480.png!
> {code:java}
> buffers = {ByteBuffer[5]@3270}
> 0 = {HeapByteBuffer@3430} "java.nio.HeapByteBuffer[pos=65536 lim=65536
> cap=65536]"
> 1 = {HeapByteBuffer@3434} "java.nio.HeapByteBuffer[pos=65536 lim=65536
> cap=65536]"
> 2 = {HeapByteBuffer@3438} "java.nio.HeapByteBuffer[pos=65536 lim=65536
> cap=65536]"
> 3 = {HeapByteBuffer@3504} "java.nio.HeapByteBuffer[pos=0 lim=65536
> cap=65536]"
> hb = {byte[65536]@3511} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, +65,436 more]
> buffers[this.dataBlkNum + ixx].equals(outputs[ixx] =true ?
> outputs = {ByteBuffer[2]@3271}
> 0 = {HeapByteBuffer@3455} "java.nio.HeapByteBuffer[pos=0 lim=65536
> cap=65536]"
> hb = {byte[65536]@3459} [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, +65,436 more]{code}
> Can this situation be judged as an anomaly?
>
> check orc file
> {code:java}
> Structure for skip_ip/_skip_file File Version: 0.12 with ORC_517 by ORC Java
> Exception in thread "main" java.io.IOException: Problem opening stripe 0
> footer in skip_ip/_skip_file. at
> org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:360)
> at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:879) at
> org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:873) at
> org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:345) at
> org.apache.orc.tools.FileDump.printMetaData(FileDump.java:276) at
> org.apache.orc.tools.FileDump.main(FileDump.java:137) at
> org.apache.orc.tools.Driver.main(Driver.java:124) Caused by:
> java.lang.IllegalArgumentException: Buffer size too small. size = 131072
> needed = 7752508 in column 3 kind LENGTH at
> org.apache.orc.impl.InStream$CompressedStream.readHeader(InStream.java:481)
> at
> org.apache.orc.impl.InStream$CompressedStream.ensureUncompressed(InStream.java:528)
> at
> org.apache.orc.impl.InStream$CompressedStream.read(InStream.java:507)
> at
> org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:59)
> at
> org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:333)
> at
> org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.readDictionaryLengthStream(TreeReaderFactory.java:2221)
> at
> org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.startStripe(TreeReaderFactory.java:2201)
> at
> org.apache.orc.impl.TreeReaderFactory$StringTreeReader.startStripe(TreeReaderFactory.java:1943)
> at
> org.apache.orc.impl.reader.tree.StructBatchReader.startStripe(StructBatchReader.java:112)
> at
> org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1251)
> at
> org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1290)
> at
> org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1333)
> at
> org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:355)
> ... 6 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
