[jira] [Updated] (HADOOP-5732) SFTP FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5732: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) +1 I committed this. Thanks Ram and Inigo > SFTP FileSystem > --- > > Key: HADOOP-5732 > URL: https://issues.apache.org/jira/browse/HADOOP-5732 > Project: Hadoop Common > Issue Type: New Feature > Components: fs > Environment: Any environment >Reporter: Íñigo Goiri >Assignee: ramtin >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-5732.008.patch, HADOOP-5732.009.patch, > HADOOP-5732.010.patch, HADOOP-5732.patch, HADOOP-5732.patch, > HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, > ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > I have implemented a FileSystem that supports SFTP. It uses JSch > (http://www.jcraft.com/jsch/) in order to manage SFTP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-5732) SFTP FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-5732:
--
Attachment: HADOOP-5732.010.patch
- Fixed {{SFTPInputStream::close()}} to be idempotent as
[required|http://docs.oracle.com/javase/7/docs/api/java/io/Closeable.html]
- Set exception as cause, rather than {{StringUtils::stringifyException}} as
message
- Changed test to use JUnit4 annotations
- Changed test to use test dir instead of {{/tmp}}
- Added an assumption that skips running the tests on Windows (haven't tested)
- Added checks to some tests to check the result of SFTP on the local fs
[~ramtinb], could you look over the patch? While there are other parts of the
impl that could be fussed over, if this has been working in production
environments then we can leave it alone.
> SFTP FileSystem
> ---
>
> Key: HADOOP-5732
> URL: https://issues.apache.org/jira/browse/HADOOP-5732
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Environment: Any environment
>Reporter: Íñigo Goiri
>Assignee: ramtin
>Priority: Minor
> Attachments: HADOOP-5732.008.patch, HADOOP-5732.009.patch,
> HADOOP-5732.010.patch, HADOOP-5732.patch, HADOOP-5732.patch,
> HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch,
> ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> I have implemented a FileSystem that supports SFTP. It uses JSch
> (http://www.jcraft.com/jsch/) in order to manage SFTP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-5732) SFTP FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin updated HADOOP-5732: --- Status: Patch Available (was: Open) > SFTP FileSystem > --- > > Key: HADOOP-5732 > URL: https://issues.apache.org/jira/browse/HADOOP-5732 > Project: Hadoop Common > Issue Type: New Feature > Components: fs > Environment: Any environment >Reporter: Íñigo Goiri >Assignee: ramtin >Priority: Minor > Attachments: HADOOP-5732.008.patch, HADOOP-5732.009.patch, > HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, > HADOOP-5732.patch, ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > I have implemented a FileSystem that supports SFTP. It uses JSch > (http://www.jcraft.com/jsch/) in order to manage SFTP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-5732) SFTP FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramtin updated HADOOP-5732:
---
Attachment: HADOOP-5732.009.patch
Thanks [~chris.douglas] for reviewing the patch.
In the new patch following items are fixed:
- Using MINA SSHD and implementing [the
suggested|https://issues.apache.org/jira/browse/HADOOP-5732?focusedCommentId=12706412&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12706412]
unit tests.
- Changing {{SFTPConnectionPool}} and {{SFTPInputStream}} access modifiers.
- Throwing {{IOException}} rather than a {{RuntimeException(SFTPException)}}.
> SFTP FileSystem
> ---
>
> Key: HADOOP-5732
> URL: https://issues.apache.org/jira/browse/HADOOP-5732
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Environment: Any environment
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Attachments: HADOOP-5732.008.patch, HADOOP-5732.009.patch,
> HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch,
> HADOOP-5732.patch, ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> I have implemented a FileSystem that supports SFTP. It uses JSch
> (http://www.jcraft.com/jsch/) in order to manage SFTP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-5732) SFTP FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5732: -- Affects Version/s: (was: 0.20.0) Status: Open (was: Patch Available) > SFTP FileSystem > --- > > Key: HADOOP-5732 > URL: https://issues.apache.org/jira/browse/HADOOP-5732 > Project: Hadoop Common > Issue Type: New Feature > Components: fs > Environment: Any environment >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Attachments: HADOOP-5732.008.patch, HADOOP-5732.patch, > HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, > ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > I have implemented a FileSystem that supports SFTP. It uses JSch > (http://www.jcraft.com/jsch/) in order to manage SFTP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-5732) SFTP FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HADOOP-5732: Status: Patch Available (was: Open) > SFTP FileSystem > --- > > Key: HADOOP-5732 > URL: https://issues.apache.org/jira/browse/HADOOP-5732 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 0.20.0 > Environment: Any environment >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Attachments: HADOOP-5732.008.patch, HADOOP-5732.patch, > HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, > ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > I have implemented a FileSystem that supports SFTP. It uses JSch > (http://www.jcraft.com/jsch/) in order to manage SFTP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-5732) SFTP FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramtin updated HADOOP-5732: --- Attachment: HADOOP-5732.008.patch A new patch for SFTP FileSystem based on JSch library. Similar to FTP FileSystem and supporting multiple concurrent connections. > SFTP FileSystem > --- > > Key: HADOOP-5732 > URL: https://issues.apache.org/jira/browse/HADOOP-5732 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 0.20.0 > Environment: Any environment >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Attachments: HADOOP-5732.008.patch, HADOOP-5732.patch, > HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, > ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > I have implemented a FileSystem that supports SFTP. It uses JSch > (http://www.jcraft.com/jsch/) in order to manage SFTP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] Updated: (HADOOP-5732) SFTP FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-5732: -- Status: Open (was: Patch Available) > SFTP FileSystem > --- > > Key: HADOOP-5732 > URL: https://issues.apache.org/jira/browse/HADOOP-5732 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 0.20.0 > Environment: Any environment >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: 0.21.0 > > Attachments: HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, > HADOOP-5732.patch, HADOOP-5732.patch, ivy-for-hadoop-7532.patch, > ivy-for-hadoop-7532.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > I have implemented a FileSystem that supports SFTP. It uses JSch > (http://www.jcraft.com/jsch/) in order to manage SFTP. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5732) SFTP FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-5732: -- Attachment: HADOOP-5732.patch I've updated the patch so that applies to trunk. As it stands the unit test isn't self-contained. Can this be done by having an embedded SFTP server like TestFTPFileSystem does? (Although see https://issues.apache.org/jira/browse/HDFS-441.) If this is not possible then the test should be renamed SFTPFileSystemTest so it isn't run as a part of the automated test suite. Ideally, the FileSystem interface contract should also be tested by subclassing FileSystemContractBaseTest and overriding the setUp() method to set the filesystem under test to be SFTPFileSystem. See TestHDFSFileSystemContract in the HDFS project for an example of how to do this. This test checks lots of edge cases and will help flush out common issues with writing FileSystem implementations. > should I add a modification in hadoop-default.xml in order to add > fs.impl.sftp value? Yes please. > SFTP FileSystem > --- > > Key: HADOOP-5732 > URL: https://issues.apache.org/jira/browse/HADOOP-5732 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 0.20.0 > Environment: Any environment >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Fix For: 0.21.0 > > Attachments: HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, > HADOOP-5732.patch, HADOOP-5732.patch, ivy-for-hadoop-7532.patch, > ivy-for-hadoop-7532.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > I have implemented a FileSystem that supports SFTP. It uses JSch > (http://www.jcraft.com/jsch/) in order to manage SFTP. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5732) SFTP FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5732: -- Status: Patch Available (was: Open) Running patch through hudson > SFTP FileSystem > --- > > Key: HADOOP-5732 > URL: https://issues.apache.org/jira/browse/HADOOP-5732 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 0.20.0 > Environment: Any environment >Reporter: Íñigo Goiri >Priority: Minor > Fix For: 0.21.0 > > Attachments: HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, > HADOOP-5732.patch, ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > I have implemented a FileSystem that supports SFTP. It uses JSch > (http://www.jcraft.com/jsch/) in order to manage SFTP. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5732) SFTP FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-5732: -- Status: Open (was: Patch Available) > SFTP FileSystem > --- > > Key: HADOOP-5732 > URL: https://issues.apache.org/jira/browse/HADOOP-5732 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 0.20.0 > Environment: Any environment >Reporter: Íñigo Goiri >Priority: Minor > Fix For: 0.21.0 > > Attachments: HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.patch, > HADOOP-5732.patch, ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > I have implemented a FileSystem that supports SFTP. It uses JSch > (http://www.jcraft.com/jsch/) in order to manage SFTP. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
