[
https://issues.apache.org/jira/browse/HIVE-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-2056:
------------------------------------------
Attachment: patch-2056.txt
Attached patch generates a single M/R job for multi group by query with
non-null common group by key set. Added configuration
hive.multigroupby.singlemr to turn on and off the optimization.
It considers no-distinct or single common distinct expression; did not multi
distinct expressions yet. Will do in a follow up if required.
Performance numbers:
||Number of rows in table|| Query || Time taken by 3 M/R jobs plan || Time
taken by Single M/R job plan||
|100 | query1| 58.416 seconds |22.099 seconds|
|33682 million | query1 | Did not succeed | 11434.308 seconds|
|33682 million | query2 | 2hrs, 48mins, 15sec |16mins, 3sec.|
Query1 did not succeed with 33682 million row table with existing plan.
Reducers failed with OOM after 12 hours. I tried many combinations of number of
reducers and Xmx values, but in vain.
Verified the correctness for 100 row table row by row; and number of rows in
the result for 33682 million rows table.
> Generate single MR job for multi groupby query.
> -----------------------------------------------
>
> Key: HIVE-2056
> URL: https://issues.apache.org/jira/browse/HIVE-2056
> Project: Hive
> Issue Type: Improvement
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Attachments: patch-2056.txt
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
