[ https://issues.apache.org/jira/browse/SPARK-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059675#comment-15059675 ]
Sean Owen commented on SPARK-12353: ----------------------------------- Yeah, since the implementation is {code} return self.map(lambda x: (x, None)).reduceByKey(lambda x, y: None).count() {code} it does seem like this is actually an implementation of countDistinct or something similar. > wrong output for countByValue and countByValueAndWindow > ------------------------------------------------------- > > Key: SPARK-12353 > URL: https://issues.apache.org/jira/browse/SPARK-12353 > Project: Spark > Issue Type: Bug > Components: Documentation, Input/Output, PySpark, Streaming > Affects Versions: 1.5.2 > Environment: Ubuntu 14.04, Python 2.7.6 > Reporter: Bo Jin > Labels: easyfix > Original Estimate: 2h > Remaining Estimate: 2h > > http://stackoverflow.com/q/34114585/4698425 > In PySpark Streaming, function countByValue and countByValueAndWindow return > one single number which is the count of distinct elements, instead of a list > of (k,v) pairs. > It's inconsistent with the documentation: > countByValue: When called on a DStream of elements of type K, return a new > DStream of (K, Long) pairs where the value of each key is its frequency in > each RDD of the source DStream. > countByValueAndWindow: When called on a DStream of (K, V) pairs, returns a > new DStream of (K, Long) pairs where the value of each key is its frequency > within a sliding window. Like in reduceByKeyAndWindow, the number of reduce > tasks is configurable through an optional argument. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org