[
https://issues.apache.org/jira/browse/HADOOP-14596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066474#comment-16066474
]
Steve Loughran commented on HADOOP-14596:
-----------------------------------------
really hard to see from the docs of what's going on with the skip. Think maybe
it should be pulled out into something self contained
Anyway, it needs to be skip(remaining + 1) to make the message go away
with {{skip(remaining)}}
{code}
2017-06-28 12:53:00,290 [ScalaTest-main-running-S3ACSVReadSuite] INFO
s3.S3ACSVReadSuite (Logging.scala:logInfo(54)) - ReadBytesReturned
Read in blocks and assess their size and duration.
This is to identify buffering quirks.
-------------------------------------------
2017-06-28 12:53:00,996 [ScalaTest-main-running-S3ACSVReadSuite] INFO
s3a.S3AFileSystem (S3AFileSystem.java:initialize(276)) - Magic committer is
enabled
2017-06-28 12:53:02,463 [ScalaTest-main-running-S3ACSVReadSuite] WARN
internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not
all bytes were read from the S3ObjectInputStream, aborting HTTP connection.
This is likely an error and may result in sub-optimal behavior. Request only
the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:04,043 [ScalaTest-main-running-S3ACSVReadSuite] WARN
internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not
all bytes were read from the S3ObjectInputStream, aborting HTTP connection.
This is likely an error and may result in sub-optimal behavior. Request only
the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:05,631 [ScalaTest-main-running-S3ACSVReadSuite] WARN
internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not
all bytes were read from the S3ObjectInputStream, aborting HTTP connection.
This is likely an error and may result in sub-optimal behavior. Request only
the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:07,203 [ScalaTest-main-running-S3ACSVReadSuite] WARN
internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not
all bytes were read from the S3ObjectInputStream, aborting HTTP connection.
This is likely an error and may result in sub-optimal behavior. Request only
the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:08,718 [ScalaTest-main-running-S3ACSVReadSuite] WARN
internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not
all bytes were read from the S3ObjectInputStream, aborting HTTP connection.
This is likely an error and may result in sub-optimal behavior. Request only
the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:10,286 [ScalaTest-main-running-S3ACSVReadSuite] WARN
internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not
all bytes were read from the S3ObjectInputStream, aborting HTTP connection.
This is likely an error and may result in sub-optimal behavior. Request only
the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:11,863 [ScalaTest-main-running-S3ACSVReadSuite] WARN
internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not
all bytes were read from the S3ObjectInputStream, aborting HTTP connection.
This is likely an error and may result in sub-optimal behavior. Request only
the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:13,386 [ScalaTest-main-running-S3ACSVReadSuite] WARN
internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not
all bytes were read from the S3ObjectInputStream, aborting HTTP connection.
This is likely an error and may result in sub-optimal behavior. Request only
the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:14,937 [ScalaTest-main-running-S3ACSVReadSuite] WARN
internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not
all bytes were read from the S3ObjectInputStream, aborting HTTP connection.
This is likely an error and may result in sub-optimal behavior. Request only
the bytes you need via a ranged GET or drain the input stream after use.
2017-06-28 12:53:16,489 [ScalaTest-main-running-S3ACSVReadSuite] WARN
internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) - Not
all bytes were read from the S3ObjectInputStream, aborting HTTP connection.
This is likely an error and may result in sub-optimal behavior. Request only
the bytes you need via a ranged GET or drain the input stream after use.
...etc
[ScalaTest-main-running-S3ACSVReadSuite] INFO s3.S3ACSVReadSuite
(Logging.scala:logInfo(54)) - 5566 blocks of size 8192;
total #of read operations 10760;
total read time=266,711,826,528 nS;
5849 ns/byte
{code}
With {{skip(remaining+1)}}
{code}
2017-06-28 13:06:13,751 [ScalaTest-main-running-S3ACSVReadSuite] INFO
s3.S3ACSVReadSuite (Logging.scala:logInfo(54)) - ReadBytesReturned
Read in blocks and assess their size and duration.
This is to identify buffering quirks.
-------------------------------------------
2017-06-28 13:06:14,451 [ScalaTest-main-running-S3ACSVReadSuite] INFO
s3a.S3AFileSystem (S3AFileSystem.java:initialize(276)) - Magic committer is
enabled
2017-06-28 13:07:17,899 [ScalaTest-main-running-S3ACSVReadSuite] INFO
s3.S3ACSVReadSuite (Logging.scala:logInfo(54)) - 5566 blocks of size 8192;
total #of read operations 10810;
total read time=63,028,617,716 nS;
1382 ns/byte
2017-06-28 13:07:17,899 [ScalaTest-main-running-S3ACSVReadSuite] INFO
s3.S3ACSVReadSuite (Logging.scala:logInfo(54)) - Read sizes
2017-06-28 13:07:17,902 [ScalaTest-main-running-S3ACSVReadSuite] INFO
s3.S3ACSVReadSuite (Logging.scala:logInfo(54)) - [7] count = 8 average duration
= 573 nS nS/byte = 81
2017-06-28 13:07:17,903 [ScalaTest-main-running-S3ACSVReadSuite] INFO
s3.S3ACSVReadSuite (Logging.scala:logInfo(54)) - [10] count = 4 average
duration = 1,959,862 nS nS/byte = 195986
2017-06-28 13:07:17,903 [ScalaTest-main-running-S3ACSVReadSuite] INFO
s3.S3ACSVReadSuite (Logging.scala:logInfo(54)) - [12] count = 2 average
duration = 1,410,564 nS nS/byte = 117547
{code}
That's an interesting fact. Maybe our skip logic is itself off by one.
> AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error
> --------------------------------------------------------------------
>
> Key: HADOOP-14596
> URL: https://issues.apache.org/jira/browse/HADOOP-14596
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-14596-001.patch, testlog.txt
>
>
> The latest SDK now tells us off when we do a seek() by aborting the TCP stream
> {code}
> - Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-06-27 15:47:35,789 [ScalaTest-main-running-S3ACSVReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-06-27 15:47:37,409 [ScalaTest-main-running-S3ACSVReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-06-27 15:47:39,003 [ScalaTest-main-running-S3ACSVReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-06-27 15:47:40,627 [ScalaTest-main-running-S3ACSVReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
