[
https://issues.apache.org/jira/browse/SQOOP-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972037#comment-13972037
]
Rodrigo Matihara commented on SQOOP-1306:
-----------------------------------------
I had to code this to my project, so, on append() method of AppendUtils class,
I did this:
// Creating a fs for each dir
FileSystem tempDirFs = tempDir.getFileSystem(options.getConf());
FileSystem userDestDirFs = userDestDir.getFileSystem(options.getConf());
// If has the same fs, use default moveFiles
if (userDestDirFs.getClass().equals(tempDirFs.getClass())) {
moveFiles(tempDirFs, tempDir, userDestDir, nextPartition);
} else {
// move files from different file systems
moveFiles(tempDirFs, tempDir, userDestDirFs, userDestDir, nextPartition);
}
The second moveFiles, creates files on target-dir using FSDataInputStream and
FSDataOutputStream and deletes temp files when it finish.
I don't know if it is the best way to do this, but it's working now.
> Allow Sqoop to move files from different FileSystems on incremental import
> --------------------------------------------------------------------------
>
> Key: SQOOP-1306
> URL: https://issues.apache.org/jira/browse/SQOOP-1306
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.4
> Reporter: Rodrigo Matihara
> Attachments: scenarios.png
>
>
> Sqoop doesn't allow to move files from different FileSystems executing an
> import (--append or --incremental).
> Trying to import using a local temp-dir and a S3 target-dir, the operation is
> complete, however the file is not created in S3 bucket and this warning is
> raised:
> WARN - Cannot append files to target dir; no such directory:
> _sqoop/15151724000000436_31417_localhost.localdomain_<tablename>
> Looking into on source, I found that AppendUtils.java considers that tempDir
> and targetDir are at the same Filesystem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
