Byron created SQOOP-1086:
----------------------------
Summary: Running multiple incremental sqoop jobs in parallel
resets the first sqoop job's --last-value
Key: SQOOP-1086
URL: https://issues.apache.org/jira/browse/SQOOP-1086
Project: Sqoop
Issue Type: Bug
Affects Versions: 1.4.0-incubating
Environment: Ubuntu 12.04.2
Reporter: Byron
Priority: Critical
I've created 2 jobs (different names) that pull from the same database(MSSQL),
but 2 different tables.
They both use incremental append.
If I run the jobs in sequence, I got no issue and the meta store for both jobs
remembers the --last-value per job.
If I run the jobs in parallel, when the 1st job finished the meta is updated
with the --last-value correctly, but once the 2nd job finished the 1st job's
meta --last-value is reset.
First Job
# create the import job into the incremental table
$ENV_SQOOP_HOME/bin/sqoop job -D mapred.job.name="Job 1" --create
"import-events" -- import --connect "$ENV_TRACKING_CONNECTION" --table
"$TABLE1" --split-by "dtmDBDateTime" --target-dir "$OUTPUT1" --incremental
append --check-column "dtmDBDateTime" --last-value "2012-01-01 00:00:00.000"
--fields-terminated-by \\t --null-string '' --null-non-string '';
Second Job
# create the import job into the table
$ENV_SQOOP_HOME/bin/sqoop job -D mapred.job.name="Job 2" --create
"import-impressions" -- import --connect "$ENV_TRACKING_CONNECTION" --table
"$TABLE2" --split-by "dtmDBDateTime" --target-dir "$OUTPUT2" --incremental
append --check-column "dtmDBDateTime" --last-value "2012-01-01 00:00:00.000"
--fields-terminated-by \\t --null-string '' --null-non-string '';
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
