[jira] [Commented] (HADOOP-18175) test failures with prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525618#comment-17525618 ] Monthon Klongklaew commented on HADOOP-18175: - assigning this to myself as I've been looking at these tests. will open a PR for it soon. > test failures with prefetching s3a input stream > --- > > Key: HADOOP-18175 > URL: https://issues.apache.org/jira/browse/HADOOP-18175 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Major > > identify and fix all test regressions from the prefetching s3a input stream -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18175) test failures with prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17516870#comment-17516870
]
Steve Loughran commented on HADOOP-18175:
-
code}
[ERROR]
testStatistics(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics)
Time elapsed: 1.591 s <<< ERROR!
java.io.EOFException: Attempted to seek or read past the end of the file 512
at
org.apache.hadoop.fs.s3a.read.S3InputStream.throwIfInvalidSeek(S3InputStream.java:440)
at
org.apache.hadoop.fs.s3a.read.S3InputStream.seek(S3InputStream.java:235)
at
org.apache.hadoop.fs.s3a.read.S3PrefetchingInputStream.seek(S3PrefetchingInputStream.java:162)
at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:85)
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:98)
at
org.apache.hadoop.fs.FCStatisticsBaseTest.testStatistics(FCStatisticsBaseTest.java:107)
{code}
it's happening in the seek(oldPos) code in the default implementation of
positioned read, because the previous unpositioned read
{code}
byte[] buf = new byte[blockSize];
int bytesRead = fstr.read(buf, 0, blockSize);
fstr.read(0, buf, 0, blockSize);
{code}
This is quite a good little failure; I'm impressed but who ever managed to
think at this corner case. pos is at EOF, so returning to it is raising the
EOF.
We will probably want seek() to not do the checks, and have read() do the
validation, which is mostly what s3a does , as it only rejects negative offsets.
> test failures with prefetching s3a input stream
> ---
>
> Key: HADOOP-18175
> URL: https://issues.apache.org/jira/browse/HADOOP-18175
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Major
>
> identify and fix all test regressions from the prefetching s3a input stream
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18175) test failures with prefetching s3a input stream
[ https://issues.apache.org/jira/browse/HADOOP-18175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513511#comment-17513511 ] Steve Loughran commented on HADOOP-18175: - looking at the unbuffer failure, all other stores return the last active read position after close() * the new stream needs to do this * API spec to say MUST return something, SHOULD return next read pos if the file hadn't been closed. if using unbuffer, MUST be that value > test failures with prefetching s3a input stream > --- > > Key: HADOOP-18175 > URL: https://issues.apache.org/jira/browse/HADOOP-18175 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Major > > identify and fix all test regressions from the prefetching s3a input stream -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18175) test failures with prefetching s3a input stream
[
https://issues.apache.org/jira/browse/HADOOP-18175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512578#comment-17512578
]
Steve Loughran commented on HADOOP-18175:
-
h3. ITestS3AContractUnbuffer
fix: getPos() on a closed file to return 0;
{code}
[ERROR]
testUnbufferOnClosedFile(org.apache.hadoop.fs.contract.s3a.ITestS3AContractUnbuffer)
Time elapsed: 0.833 s <<< ERROR!
java.io.IOException: Stream is closed!
at
org.apache.hadoop.fs.s3a.read.S3PrefetchingInputStream.throwIfClosed(S3PrefetchingInputStream.java:224)
at
org.apache.hadoop.fs.s3a.read.S3PrefetchingInputStream.getPos(S3PrefetchingInputStream.java:105)
at
org.apache.hadoop.fs.FSDataInputStream.getPos(FSDataInputStream.java:81)
at
org.apache.hadoop.fs.contract.AbstractContractUnbufferTest.unbuffer(AbstractContractUnbufferTest.java:124)
at
org.apache.hadoop.fs.contract.AbstractContractUnbufferTest.testUnbufferOnClosedFile(AbstractContractUnbufferTest.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}
h3. ITestS3AUnbuffer
instanceof and statistics assertions failing.
{code}
[ERROR] testUnbuffer(org.apache.hadoop.fs.s3a.ITestS3AUnbuffer) Time elapsed:
2.385 s <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at
org.apache.hadoop.fs.s3a.ITestS3AUnbuffer.testUnbuffer(ITestS3AUnbuffer.java:75)
[ERROR] testUnbufferStreamStatistics(org.apache.hadoop.fs.s3a.ITestS3AUnbuffer)
Time elapsed: 1.115 s <<< FAILURE!
java.lang.AssertionError: Count of stream_read_bytes starting=0 current=0
diff=0: stream_read_bytes expected:<8> but was:<0>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at
org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:998)
at
org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:1007)
at
org.apache.hadoop.fs.s3a.ITestS3AUnbuffer.testUnbufferStreamStatistics(ITestS3AUnbuffer.java:149)
{code}
h3. ITestS3ARequesterPays statistics assertion.
{code}
[ERROR]
testRequesterPaysOptionSuccess(org.apache.hadoop.fs.s3a.ITestS3ARequesterPays)
Time elapsed: 2.198 s <<< FAILURE!
java.lang.AssertionError:
[Counter named stream_read_opened]
Expecting:
<0L>
to be greater than:
<1L>
at
org.apache.hadoop.fs.s3a.ITestS3ARequesterPays.testRequesterPaysOptionSuccess(ITestS3ARequesterPays.java:78)
{code}
fix: wire up the statistics and/or change assertion based on stream type
h3.
testStatistics(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics)
T
{code}
java.io.EOFException: Attempted to seek or read past the end of the file 512
at
org.apache.hadoop.fs.s3a.read.S3InputStream.throwIfInvalidSeek(S3InputStream.java:439)
at
org.apache.hadoop.fs.s3a.read.S3InputStream.seek(S3InputStream.java:234)
at
org.apache.hadoop.fs.s3a.read.S3PrefetchingInputStream.seek(S3PrefetchingInputStream.java:161)
at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:85)
at
org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:98)
at
org.apache.hadoop.fs.FCStatisticsBaseTest.testStatistics(FCStatisticsBaseTest.java:107)
{code}
> test failures with prefetching s3a input stream
> ---
>
> Key: HADOOP-18175
> URL: https://issues.apache.org/jira/browse/HADOOP-18175
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Priority: Major
>
> identify and fix all test regressions from the prefetching s3a input stream
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
