[ 
https://issues.apache.org/jira/browse/HIVE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong updated HIVE-3640:
--------------------------------

    Component/s: Query Processor
    
> Reducer allocation is incorrect if enforce bucketing and mapred.reduce.tasks 
> are both set
> -----------------------------------------------------------------------------------------
>
>                 Key: HIVE-3640
>                 URL: https://issues.apache.org/jira/browse/HIVE-3640
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.10.0
>            Reporter: Vighnesh Avadhani
>            Assignee: Vighnesh Avadhani
>            Priority: Minor
>             Fix For: 0.10.0
>
>         Attachments: HIVE-3640.1.patch.txt
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> When I enforce bucketing and fix the number of reducers via 
> mapred.reduce.tasks Hive ignores my input and instead takes the largest value 
> <= hive.exec.reducers.max that is also an even divisor of num_buckets. In 
> other words, if I set 1024 buckets and set mapred.reduce.tasks=1024 I'll get. 
> . . 256 reducers. If I set 1997 buckets and set mapred.reduce.tasks=1997 I'll 
> get. . . 1 reducer. 
> This is totally crazy, and it's far, far crazier when the data inputs get 
> large. In the latter case the bucketing job will almost certainly fail 
> because we'll most likely try to stuff several TB of input through a single 
> reducer. We'll also drastically reduce the effectiveness of bucketing, since 
> the buckets themselves will be larger.
> If the user sets mapred.reduce.tasks in a query that inserts into a bucketed 
> table we should either accept that value or raise an exception if it's 
> invalid relative to the number of buckets. We should absolutely NOT override 
> the user's direction and fall back on automatically allocating reducers based 
> on some obscure logic dictated by completely different setting. 
> I have yet to encounter a single person who expected this the first time, so 
> it's clearly a bug.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to