[
https://issues.apache.org/jira/browse/HADOOP-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807918#comment-16807918
]
Sahil Takiar commented on HADOOP-14747:
---------------------------------------
Assigning to myself as I am planning to work on this. I already have a patch
tested and ready to go. I'm currently working on writing some unit tests. My
unit test plan is to:
* Write one unit test that is purely mock based and validates that calling
{{unbuffer}} actually closes the underlying stream
* Write another itest that actually calls {{unbuffer}} against S3; expose a
package-private method in {{S3AInputStream}} called {{isObjectStreamOpen}} that
checks if the {{wrappedStream}} is {{null}} or not
Before I post anything, have a few questions that I'm hoping
[~ste...@apache.org] can help me with:
{quote}Contract specification of unbuffer
{quote}
When you say specification, do you mean javadocs for
{{S3AInputStream#unbuffer}}? Or is there a written specification somewhere
else?
{quote}Contract test for filesystems which declare their support
{quote}
I'm guessing this is related to {{ITestS3AContract}} tests? I don't see a
{{AbstractContractUnbufferTest}} so I'm guessing we will have to add one? Off
the top of my head, I would think the contract tests should validate:
* If {{unbuffer}} is called on a closed file, throw {{IOException: Stream
closed}}
* {{unbuffer}} can be called both before and after a call to {{read}}
* {{unbuffer}} can be called on an empty file
* {{unbuffer}} can be called multiple times, both with and without
intermingled reads
> S3AInputStream to implement CanUnbuffer
> ---------------------------------------
>
> Key: HADOOP-14747
> URL: https://issues.apache.org/jira/browse/HADOOP-14747
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.1
> Reporter: Steve Loughran
> Assignee: Sahil Takiar
> Priority: Major
>
> HBase relies on FileSystems implementing {{CanUnbuffer.unbuffer()}} to force
> input streams to free up remote connections (HBASE-9393). This works for
> HDFS, but not elsewhere.
> S3A input stream can implement {{CanUnbuffer.unbuffer()}} by closing the
> input stream and relying on lazy seek to reopen it on demand.
> Needs
> * Contract specification of unbuffer. As in "who added a new feature to
> filesystems but forgot to mention what it should do?"
> * Contract test for filesystems which declare their support.
> * S3AInputStream to call {{closeStream()}} on a call to {{unbuffer()}}.
> * Test case
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
