[
https://issues.apache.org/jira/browse/HADOOP-15358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HADOOP-15358:
----------------------------------
Fix Version/s: 3.3.0
> SFTPConnectionPool connections leakage
> --------------------------------------
>
> Key: HADOOP-15358
> URL: https://issues.apache.org/jira/browse/HADOOP-15358
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 3.0.0
> Reporter: Mikhail Pryakhin
> Assignee: Mikhail Pryakhin
> Priority: Critical
> Fix For: 3.3.0
>
> Attachments: HADOOP-15358.001.patch
>
>
> Methods of SFTPFileSystem operate on poolable ChannelSftp instances, thus
> some methods of SFTPFileSystem are chained together resulting in establishing
> multiple connections to the SFTP server to accomplish one compound action,
> those methods are listed below:
> # mkdirs method
> the public mkdirs method acquires a new ChannelSftp from the pool [1]
> and then recursively creates directories, checking for the directory
> existence beforehand by calling the method exists[2] which delegates to the
> getFileStatus(ChannelSftp channel, Path file) method [3] and so on until it
> ends up in returning the FilesStatus instance [4]. The resource leakage
> occurs in the method getWorkingDirectory which calls the getHomeDirectory
> method [5] which in turn establishes a new connection to the sftp server
> instead of using an already created connection. As the mkdirs method is
> recursive this results in creating a huge number of connections.
> # open method [6]. This method returns an instance of FSDataInputStream
> which consumes SFTPInputStream instance which doesn't return an acquired
> ChannelSftp instance back to the pool but instead it closes it[7]. This leads
> to establishing another connection to an SFTP server when the next method is
> called on the FileSystem instance.
> [1]
> https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L658
> [2]
> https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L321
> [3]
> https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L202
> [4]
> https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L290
> [5]
> https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L640
> [6]
> https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L504
> [7]
> https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java#L123
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
