[
https://issues.apache.org/jira/browse/HADOOP-14596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066278#comment-16066278
]
Steve Loughran commented on HADOOP-14596:
-----------------------------------------
It turns out the problem is that on 1.11+ the close() call doesn't close the
stream any more if there is outstanding data (good, stops the "read to the end
of the 20 GB file" problem, but warns, even if the readahead value is low
enough for us to consider harmful. That is: if there is >0 bytes in the stream,
it calls abort().
This means that low-cost close operations have just become expensive ones; a
critical change
# S3A input stream *must* read to the end of the current request before calling
close() on the underlying stream. a skip() should suffice
# leave log settings alone
# do we cover error message in docs? Ideally it should go away. Maybe: if you
see this, get in touch.
> AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error
> --------------------------------------------------------------------
>
> Key: HADOOP-14596
> URL: https://issues.apache.org/jira/browse/HADOOP-14596
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Steve Loughran
> Priority: Minor
> Attachments: testlog.txt
>
>
> The latest SDK now tells us off when we do a seek() by aborting the TCP stream
> {code}
> - Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-06-27 15:47:35,789 [ScalaTest-main-running-S3ACSVReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-06-27 15:47:37,409 [ScalaTest-main-running-S3ACSVReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-06-27 15:47:39,003 [ScalaTest-main-running-S3ACSVReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-06-27 15:47:40,627 [ScalaTest-main-running-S3ACSVReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
