[
https://issues.apache.org/jira/browse/HIVE-339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680984#action_12680984
]
Namit Jain commented on HIVE-339:
---------------------------------
Say the query is: select col2, count(distinct col2) from T group by col1;
the mapper emits:
col1 col2 col3
Key 19 1
Key 19 1
Key 22 1
Key 22 1
Key 23 1
The expected answer is:
Key 3
So, you walk over the key+sort (which is col1 + col2), which makes sure that in
the above example, 2nd and 4th rows are ignored,
and then iterate is called 3 times, which result in (key 3)
However,if you called merge in the reducer: it would result in:
key (19+22+23) --> key 64
The same should work for avg also. I am running a query on mobile_mms for
february currently to validate.
Partial results do not mean anything for distincts whereas they can be
aggregated for non-distincts.
> [Hive] problem in count distinct in 1mapreduce job with map side aggregation
> ----------------------------------------------------------------------------
>
> Key: HIVE-339
> URL: https://issues.apache.org/jira/browse/HIVE-339
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.339.1.patch, hive.339.2.patch, hive.339.3.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.