[
https://issues.apache.org/jira/browse/MAPREDUCE-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Kimball updated MAPREDUCE-1489:
-------------------------------------
Attachment: MAPREDUCE-1489.patch
Attaching patch which improves DataDrivenDBInputFormat in this manner. Also
updated use of deprecated config keys nearby.
The majority of tests for DDDBIF are implicit in the Sqoop tests; I've modified
these so that some Sqoop tests respect the old code-path by requesting multiple
map tasks, but other Sqoop tests will use a single task.
This patch is based on the code in MAPREDUCE-1460; will mark as patch-available
when that's committed.
> DataDrivenDBInputFormat should not query the database when generating only
> one split
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1489
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1489
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/sqoop
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: MAPREDUCE-1489.patch
>
>
> DataDrivenDBInputFormat runs a query to establish bounding values for each
> split it generates; but if it's going to generate only one split
> (mapreduce.job.maps == 1), then there's no reason to do this. This will
> remove overhead associated with a single-threaded import of a non-indexed
> table since it avoids a full table scan.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
