[
https://issues.apache.org/jira/browse/SQOOP-603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459844#comment-13459844
]
Cheolsoo Park commented on SQOOP-603:
-------------------------------------
+1. I will commit after running tests.
> Support small intervals in IntegerSplitter implementation
> ---------------------------------------------------------
>
> Key: SQOOP-603
> URL: https://issues.apache.org/jira/browse/SQOOP-603
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Jarek Jarcec Cecho
> Assignee: Jarek Jarcec Cecho
> Fix For: 1.4.3
>
> Attachments: SQOOP-603.patch
>
>
> IntegerSplitter is currently creating splits of following nature:
> {code}
> minimal value <= x < splitPoint1
> splitPoint1 <= x < splitPoint2
> ...
> splitPointN <= x <= maximal value
> {code}
> Please notice that upper bound is always with using condition "<" with
> exception of the last split that is using condition "<=". This is perfectly
> fine when creating reasonable amount of splits on very huge interval.
> This approach will however cause issues on very small intervals. For example
> following splits will be created on interval [0, 5] with 5 splits:
> * 0 <= x < 1
> * 1 <= x < 2
> * 2 <= x < 3
> * 3 <= x < 4
> * 4 <= x <= 5
> Notice that all splits have equal count of numbers except the last one having
> two numbers - 4 and 5. This becomes very huge issue when for example user
> needs to create one split per one partition as one mapper will end up with
> moving two partitions and thus entire job will take twice as long as the
> other ones.
> Jarcec
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
