[
https://issues.apache.org/jira/browse/SQOOP-1306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972924#comment-13972924
]
Rodrigo Matihara commented on SQOOP-1306:
-----------------------------------------
Good Venkat Ranganathan, I will try Oozie too.
But, I steel thinking that could be a good improvement on Sqoop, allow the use
of different file systems to temp dir and target dir.
Supposing that I call this:
bq. sqoop import jdbc:mysql://xxxxx:3306/sqoop --username root --password root
--table yyyyy --target-dir s3n://<bucket-name>/sqoop --incremental append
--check-column id --last-value 120
I saw this bug https://issues.apache.org/jira/browse/SQOOP-1303, that allows a
different file system on target-dir, so this command is not working on
sqoop-1.4.4, but will work on sqoop-1.4.5, its great, cause Sqoop will allow
use different FileSystems to target-dir.
But, if I call this:
bq. sqoop import jdbc:mysql://xxxxx:3306/sqoop --username root --password root
--table yyyyy --target-dir s3n://<bucket-name>/sqoop --incremental append
--check-column id --last-value 120
and supposing that *strong*sqoop.test.import.rootDir=file:/tmp/sqoop/*strong*
I will have problems, cause Sqoop will try to rename temp files that exists on
/tmp/sqoop/... to s3n://<bucket-name>/sqoop/...
I'm proposing to Sqoop allow, when temp-dir and target-dir have different
FileSystems, really moves files from temp to target.
I have coded this on a sqoop version that I'm studying, and I could send the
class to you guys check.
I'm sorry about my english, I think that it is not helping our discussion, but
my intentions are just to help the community ;P
> Allow Sqoop to move files from different FileSystems on incremental import
> --------------------------------------------------------------------------
>
> Key: SQOOP-1306
> URL: https://issues.apache.org/jira/browse/SQOOP-1306
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.4
> Reporter: Rodrigo Matihara
> Attachments: scenarios.png
>
>
> Sqoop doesn't allow to move files from different FileSystems executing an
> import (--append or --incremental).
> Trying to import using a local temp-dir and a S3 target-dir, the operation is
> complete, however the file is not created in S3 bucket and this warning is
> raised:
> WARN - Cannot append files to target dir; no such directory:
> _sqoop/15151724000000436_31417_localhost.localdomain_<tablename>
> Looking into on source, I found that AppendUtils.java considers that tempDir
> and targetDir are at the same Filesystem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
