[
https://issues.apache.org/jira/browse/SQOOP-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Lin updated SQOOP-3330:
----------------------------
Description:
When adding --append to Sqoop directory import with
-Dmapreduce.output.basename, all files will be ignored, which end up nothing
being imported. See below DEBUG output:
{code}
sqoop import -Dmapreduce.output.basename="eric-test" --connect
jdbc:mysql://mysql-host.com/test --username root --password 'root' --table test
--target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by '\t' --verbose
--append
18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00000 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00001 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00002 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder
14935e396acc4ea7b9a6236c66064c9b_test
{code}
This is due to AppendUtils only recognizes file name starts with "part.*-([0-9]"
https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46
was:
When adding --append to Sqoop directory import with
-Dmapreduce.output.basename, all files will be ignored, which end up nothing
being imported. See below DEBUG output:
sqoop import -Dmapreduce.output.basename="eric-test" --connect
jdbc:mysql://mysql-host.com/test --username root --password 'root' --table test
--target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by '\t' --verbose
--append
{code}
18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00000 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00001 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00002 ignored
18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder
14935e396acc4ea7b9a6236c66064c9b_test
{code}
This is due to AppendUtils only recognizes file name starts with "part.*-([0-9]"
https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46
> Sqoop --append does not work with -Dmapreduce.output.basename
> -------------------------------------------------------------
>
> Key: SQOOP-3330
> URL: https://issues.apache.org/jira/browse/SQOOP-3330
> Project: Sqoop
> Issue Type: Bug
> Components: tools
> Affects Versions: 1.4.7
> Reporter: Eric Lin
> Assignee: Eric Lin
> Priority: Major
>
> When adding --append to Sqoop directory import with
> -Dmapreduce.output.basename, all files will be ignored, which end up nothing
> being imported. See below DEBUG output:
> {code}
> sqoop import -Dmapreduce.output.basename="eric-test" --connect
> jdbc:mysql://mysql-host.com/test --username root --password 'root' --table
> test --target-dir /tmp/ericlin-test/sqoop/test --fields-terminated-by '\t'
> --verbose --append
> 18/05/28 22:24:44 INFO util.AppendUtils: Appending to directory test
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: _SUCCESS ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00000 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00001 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Filename: eric-test-m-00002 ignored
> 18/05/28 22:24:44 DEBUG util.AppendUtils: Deleting temporary folder
> 14935e396acc4ea7b9a6236c66064c9b_test
> {code}
> This is due to AppendUtils only recognizes file name starts with
> "part.*-([0-9]"
> https://github.com/apache/sqoop/blob/branch-1.4.7/src/java/org/apache/sqoop/util/AppendUtils.java#L46
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
