[
https://issues.apache.org/jira/browse/HDFS-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241066#comment-15241066
]
Steve Loughran commented on HDFS-10291:
---------------------------------------
1. test is doing a short circuited read of a 13 byte file
{{doTestShortCircuitRead(false, size= 13, readOffset= 0)}}
2. # creates 13 bytes of data, saves it: {{byte[] fileData =
AppendTestUtil.randomBytes(seed, size)}}
3. dest buffer is created for this
{code}
byte[] actual = new byte[expected.length-readOffset];
{code}
4 does some small reads:
{code}
//Read a small number of bytes first.
int nread = stm.read(actual, 0, 3);
nread += stm.read(actual, nread, 2);
//Read across chunk boundary
nread += stm.read(actual, nread, 517); // ** HERE **
{code}
The exception is being raised as the code is asking to read 517 bytes into a
buffer 13 bytes long. This breaks IOStream's rules: you can't ask for more than
you have space for. It says that clearly in the IOStream API spec; what was
added in HADOOP-12994 was the checking of passing in too big a length or
negative offsets.
I think this a bug in the test. Whatever it is trying to do, it shouldn't be
trying to do it on such a small buffer.
What's interesting though is when you delve into the code: the block reader
logic doesn't look at the length of the read at all. That is, it appears to
fill up the entire byte array passed in, from the offset supplied, stopping at
the end of the buffer or file, whichever comes first.
Which is something that other code (i.e. production code) could be relying on.
They shouldn't, as the code will break when working with any FS other than
HDFS, but there is a risk that they might.
What to do?
# retain checks, fix test.
# log at warning and shrink len parameter when passed down. People shouldn't be
doing this, but HDFS will reluctantly let you.
> TestShortCircuitLocalRead failing
> ---------------------------------
>
> Key: HDFS-10291
> URL: https://issues.apache.org/jira/browse/HDFS-10291
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>
> {{TestShortCircuitLocalRead}} failing as length of read is considered off end
> of buffer. There's an off-by-one error somewhere in the test or the new
> validation code
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
