[ 
https://issues.apache.org/jira/browse/HIVE-339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680984#action_12680984
 ] 

Namit Jain commented on HIVE-339:
---------------------------------

Say the query is: select col2, count(distinct col2) from T group by col1;

the mapper emits:

col1   col2  col3
Key    19   1
Key    19   1
Key    22   1
Key    22   1
Key    23   1

The expected answer is:

Key  3

So, you walk over the key+sort (which is col1 + col2), which makes sure that in 
the above example, 2nd and 4th rows are ignored,
and then iterate is called 3 times, which result in (key 3)

However,if you called merge in the reducer: it would result in:

key (19+22+23) --> key 64


The same should work for avg also. I am running a query on mobile_mms for 
february currently to validate.
Partial results do not mean anything for distincts whereas they can be 
aggregated for non-distincts.







> [Hive] problem in count distinct in 1mapreduce job with map side aggregation
> ----------------------------------------------------------------------------
>
>                 Key: HIVE-339
>                 URL: https://issues.apache.org/jira/browse/HIVE-339
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Namit Jain
>            Assignee: Namit Jain
>         Attachments: hive.339.1.patch, hive.339.2.patch, hive.339.3.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to