[
https://issues.apache.org/jira/browse/HIVE-7158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014549#comment-14014549
]
Gunther Hagleitner commented on HIVE-7158:
------------------------------------------
Patch .1 includes patch for HIVE-7121 to make it compile. Will rebase once
that's in.
> Use Tez auto-parallelism in Hive
> --------------------------------
>
> Key: HIVE-7158
> URL: https://issues.apache.org/jira/browse/HIVE-7158
> Project: Hive
> Issue Type: Bug
> Reporter: Gunther Hagleitner
> Assignee: Gunther Hagleitner
> Attachments: HIVE-7158.1.patch
>
>
> Tez can optionally sample data from a fraction of the tasks of a vertex and
> use that information to choose the number of downstream tasks for any given
> scatter gather edge.
> Hive estimates the count of reducers by looking at stats and estimates for
> each operator in the operator pipeline leading up to the reducer. However, if
> this estimate turns out to be too large, Tez can reign in the resources used
> to compute the reducer.
> It does so by combining partitions of the upstream vertex. It cannot,
> however, add reducers at this stage.
> I'm proposing to let users specify whether they want to use auto-parallelism
> or not. If they do there will be scaling factors to determine max and min
> reducers Tez can choose from. We will then partition by max reducers, letting
> Tez sample and reign in the count up until the specified min.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
