[
https://issues.apache.org/jira/browse/HADOOP-14444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074462#comment-16074462
]
Lukas Waldmann commented on HADOOP-14444:
-----------------------------------------
1) seek is necessary for interrupted transfers and it doesn't disconnect in
sense of interrupting tcp connection and logging again. In close exiting data
stream, while keeping control connection, and open it again in correct
position. Of course it's not as effective as local fs but it does the job.
2) I beg to differ. Semantics of both protocols are the same. How data are
actually transferred is only an implementation detail. You can see the common
package actually contains majority of the code
3) will try to pit it in
4) no - connection is returned to the pool and reused. For sftp it is actually
greatest time saver as initiation of ssh connection is really expensive
5) Thank you. It's always nice to see somebody is interested in work you have
done. And so far you are the only one having look at the code. :)
> New implementation of ftp and sftp filesystems
> ----------------------------------------------
>
> Key: HADOOP-14444
> URL: https://issues.apache.org/jira/browse/HADOOP-14444
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Affects Versions: 2.8.0
> Reporter: Lukas Waldmann
> Assignee: Lukas Waldmann
> Attachments: HADOOP-14444.2.patch, HADOOP-14444.3.patch,
> HADOOP-14444.4.patch, HADOOP-14444.5.patch, HADOOP-14444.patch
>
>
> Current implementation of FTP and SFTP filesystems have severe limitations
> and performance issues when dealing with high number of files. Mine patch
> solve those issues and integrate both filesystems such a way that most of the
> core functionality is common for both and therefore simplifying the
> maintainability.
> The core features:
> * Support for HTTP/SOCKS proxies
> * Support for passive FTP
> * Support of connection pooling - new connection is not created for every
> single command but reused from the pool.
> For huge number of files it shows order of magnitude performance improvement
> over not pooled connections.
> * Caching of directory trees. For ftp you always need to list whole directory
> whenever you ask information about particular file.
> Again for huge number of files it shows order of magnitude performance
> improvement over not cached connections.
> * Support of keep alive (NOOP) messages to avoid connection drops
> * Support for Unix style or regexp wildcard glob - useful for listing a
> particular files across whole directory tree
> * Support for reestablishing broken ftp data transfers - can happen
> surprisingly often
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
