[
https://issues.apache.org/jira/browse/PHOENIX-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146707#comment-16146707
]
Lars Hofhansl edited comment on PHOENIX-418 at 8/30/17 6:31 AM:
----------------------------------------------------------------
Shouldn't the clearspring library be part of the server jar? I just did a clean
build from scratch*, and indeed APPROX_COUNT_DISTINCT fails with
{code}
...
Caused by: java.lang.NoClassDefFoundError:
com/clearspring/analytics/stream/cardinality/HyperLogLogPlus
...
{code}
thrown at the server.
\* mvn -DskipTests package
was (Author: lhofhansl):
Shouldn't the clearspring library be part of the server jar? I just did a clean
build from scratch, and indeed APPROX_COUNT_DISTINCT fails with
{code}
...
Caused by: java.lang.NoClassDefFoundError:
com/clearspring/analytics/stream/cardinality/HyperLogLogPlus
...
{code}
thrown at the server.
> Support approximate COUNT DISTINCT
> ----------------------------------
>
> Key: PHOENIX-418
> URL: https://issues.apache.org/jira/browse/PHOENIX-418
> Project: Phoenix
> Issue Type: Task
> Reporter: James Taylor
> Assignee: Ethan Wang
> Labels: gsoc2016
> Attachments: PHOENIX-418-v1.patch, PHOENIX-418-v2.patch,
> PHOENIX-418-v3.patch, PHOENIX-418-v4.patch, PHOENIX-418-v5.patch,
> PHOENIX-418-v6.patch
>
>
> Support an "approximation" of count distinct to prevent having to hold on to
> all distinct values (since this will not scale well when the number of
> distinct values is huge). The Apache Drill folks have had some interesting
> discussions on this
> [here](http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/201306.mbox/%3CJIRA.12650169.1369931282407.88049.1370645900553%40arcas%3E).
> They recommend using [Welford's
> method](http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance_Online_algorithm).
> I'm open to having a config option that uses exact versus approximate. I
> don't have experience implementing an approximate implementation, so I'm not
> sure how much state is required to keep on the server and return to the
> client (other than realizing it'd be much less that returning all distinct
> values and their counts).
> Update:
> Syntax of using approximate count distinct as:
> select APPROX_COUNT_DISTINCT(name) from person
> select APPROX_COUNT_DISTINCT(address||name) from person
> It is equivalent of Select COUNT(DISTINCT ID) from person. Implemented using
> hyperloglog, see discuss below.
> Source code patch link below, co-authorred with [~swapna]
> https://git-wip-us.apache.org/repos/asf?p=phoenix.git;a=commitdiff;h=d6381afc3af976ccdbb874d4458ea17b1e8a1d32
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
