[jira] [Commented] (SPARK-19553) Add GroupedData.countApprox()
[ https://issues.apache.org/jira/browse/SPARK-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15870780#comment-15870780 ] Nicholas Chammas commented on SPARK-19553: -- The utility of 1) would be being able to count items instead of distinct items, unless I misunderstood what you're saying. I would imagine that just counting items (as opposed to distinct items) would be cheaper, in addition to being semantically different. I'll open a PR for 3), unless someone else wants to step in and do that. > Add GroupedData.countApprox() > - > > Key: SPARK-19553 > URL: https://issues.apache.org/jira/browse/SPARK-19553 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.1.0 >Reporter: Nicholas Chammas >Priority: Minor > > We already have a > [{{pyspark.sql.functions.approx_count_distinct()}}|http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.approx_count_distinct] > that can be applied to grouped data, but it seems odd that you can't just > get regular approximate count for grouped data. > I imagine the API would mirror that for > [{{RDD.countApprox()}}|http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.countApprox], > but I'm not sure: > {code} > (df > .groupBy('col1') > .countApprox(timeout=300, confidence=0.95) > .show()) > {code} > Or, if we want to mirror the {{approx_count_distinct()}} function, we can do > that too. I'd want to understand why that function doesn't take a timeout or > confidence parameter, though. Also, what does {{rsd}} mean? It's not > documented. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19553) Add GroupedData.countApprox()
[ https://issues.apache.org/jira/browse/SPARK-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864326#comment-15864326 ] Michael Armbrust commented on SPARK-19553: -- It seems like there are a couple of distinct feature requests here: 1) A different implementation of countApprox that that is based on scheduling timeouts instead of sketching to find distinct items. 2) A short hand on GroupedData that could be used to call the aforementioned function. 3) Better docs for the existing approx count distinct function. [Perhaps copied from here|https://github.com/apache/spark/blob/7a7ce272fe9a703f58b0180a9d2001ecb5c4b8db/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L235] I'm skeptical of the utility of #1, but could be convinced with performance for real work tasks (you can call {{df.rdd.countApprox()}} today). #3 seems like we should obviously do it. > Add GroupedData.countApprox() > - > > Key: SPARK-19553 > URL: https://issues.apache.org/jira/browse/SPARK-19553 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.1.0 >Reporter: Nicholas Chammas >Priority: Minor > > We already have a > [{{pyspark.sql.functions.approx_count_distinct()}}|http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.approx_count_distinct] > that can be applied to grouped data, but it seems odd that you can't just > get regular approximate count for grouped data. > I imagine the API would mirror that for > [{{RDD.countApprox()}}|http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.countApprox], > but I'm not sure: > {code} > (df > .groupBy('col1') > .countApprox(timeout=300, confidence=0.95) > .show()) > {code} > Or, if we want to mirror the {{approx_count_distinct()}} function, we can do > that too. I'd want to understand why that function doesn't take a timeout or > confidence parameter, though. Also, what does {{rsd}} mean? It's not > documented. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19553) Add GroupedData.countApprox()
[ https://issues.apache.org/jira/browse/SPARK-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864000#comment-15864000 ] Nicholas Chammas commented on SPARK-19553: -- Quick API question for you [~marmbrus]: Is this feature request appropriate? If yes, would it be better expressed as a SQL function or as a method on {{GroupedData}}? > Add GroupedData.countApprox() > - > > Key: SPARK-19553 > URL: https://issues.apache.org/jira/browse/SPARK-19553 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.1.0 >Reporter: Nicholas Chammas >Priority: Minor > > We already have a > [{{pyspark.sql.functions.approx_count_distinct()}}|http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.approx_count_distinct] > that can be applied to grouped data, but it seems odd that you can't just > get regular approximate count for grouped data. > I imagine the API would mirror that for > [{{RDD.countApprox()}}|http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.countApprox], > but I'm not sure: > {code} > (df > .groupBy('col1') > .countApprox(timeout=300, confidence=0.95) > .show()) > {code} > Or, if we want to mirror the {{approx_count_distinct()}} function, we can do > that too. I'd want to understand why that function doesn't take a timeout or > confidence parameter, though. Also, what does {{rsd}} mean? It's not > documented. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19553) Add GroupedData.countApprox()
[ https://issues.apache.org/jira/browse/SPARK-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861735#comment-15861735 ] Nicholas Chammas commented on SPARK-19553: -- I needed something like this today. I was profiling some data and didn't need exact counts. > Add GroupedData.countApprox() > - > > Key: SPARK-19553 > URL: https://issues.apache.org/jira/browse/SPARK-19553 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.1.0 >Reporter: Nicholas Chammas >Priority: Minor > > We already have a > [{{pyspark.sql.functions.approx_count_distinct()}}|http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.approx_count_distinct] > that can be applied to grouped data, but it seems odd that you can't just > get regular approximate count for grouped data. > I imagine the API would mirror that for > [{{RDD.countApprox()}}|http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.countApprox], > but I'm not sure: > {code} > (df > .groupBy('col1') > .countApprox(timeout=300, confidence=0.95) > .show()) > {code} > Or, if we want to mirror the {{approx_count_distinct()}} function, we can do > that too. I'd want to understand why that function doesn't take a timeout or > confidence parameter, though. Also, what does {{rsd}} mean? It's not > documented. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org