Yang Jie created HADOOP-20060:
---------------------------------
Summary: ChecksumFileSystem: readVectored() fails when the .crc
header is unreadable, while read() falls back to unverified reads
Key: HADOOP-20060
URL: https://issues.apache.org/jira/browse/HADOOP-20060
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Yang Jie
When a file's .crc exists but its header cannot be read (zero length,
truncated, or bad magic), ChecksumFSInputChecker logs "Problem opening checksum
file ... Ignoring exception" and reads the data without verification.
readVectored() does not do the same. It only hands the ranges to the data
stream when {{sums == null}}, and here {{sums}} is still open, so it computes
checksum ranges with the default {{bytesPerSum}} of 1 and reads them from the
broken .crc. That fails with "EOFException: Read past End of File", or with a
ChecksumException at meaningless offsets if the .crc is large.
hasCapability(VECTOREDIO_BUFFERS_SLICED) also returns true.
A zero-length .crc is what a writer leaves behind when it dies before closing
the file, because the header stays in a buffer until roughly 511 KiB of data
has been written. Parquet 1.16 enables vectored reads by default, so Spark
reading such a Parquet file from file:// reads the footer and then fails on the
row groups. With Parquet 1.15 the same file can be read.
The fix makes {{dataStreamToHandleVectorIO()}} also delegate when checksums are
not being verified ({{!needChecksum()}}), as its javadoc already says it should.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
