[
https://issues.apache.org/jira/browse/SQOOP-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979729#comment-13979729
]
Henrique Andrade commented on SQOOP-1306:
-----------------------------------------
Jarek and Venkat,
I think the functionality from Sqoop is pretty clear and Rodrigo is not talking
about adding new functionalities or change current ones. The main point here,
is cost. Right now the temp files are being written to S3 that means that we
have the cost to upload those files using Internet connection and have several
unnecessary read and write operations on S3. What Rodrigo is proposing is to
keep the temp files generated from Sqoop on the local temp directory and then
at the end of the process move the final file that to S3 as is doing right now.
The change that Rodrigo performed is just to have the option to define a local
temp dir or S3 temp dir. If the user defines a s3 temp dir, it will have the
same activities that has on the current code if the user defines and local temp
dir then the temp files will be placed on the local temp dir and at the end
will be moved to S3.
Makes sense?
> Allow Sqoop to move files from different FileSystems on incremental import
> --------------------------------------------------------------------------
>
> Key: SQOOP-1306
> URL: https://issues.apache.org/jira/browse/SQOOP-1306
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.4
> Reporter: Rodrigo Matihara
> Attachments: scenarios.png
>
>
> Sqoop doesn't allow to move files from different FileSystems executing an
> import (--append or --incremental).
> Trying to import using a local temp-dir and a S3 target-dir, the operation is
> complete, however the file is not created in S3 bucket and this warning is
> raised:
> WARN - Cannot append files to target dir; no such directory:
> _sqoop/15151724000000436_31417_localhost.localdomain_<tablename>
> Looking into on source, I found that AppendUtils.java considers that tempDir
> and targetDir are at the same Filesystem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
