[SPARK-13339][DOCS] Clarify commutative / associative operator requirements for reduce, fold
Clarify that reduce functions need to be commutative, and fold functions do not See https://github.com/apache/spark/pull/11091 Author: Sean Owen <so...@cloudera.com> Closes #11217 from srowen/SPARK-13339. Project: http://git-wip-us.apache.org/repos/asf/bahir/repo Commit: http://git-wip-us.apache.org/repos/asf/bahir/commit/5d450505 Tree: http://git-wip-us.apache.org/repos/asf/bahir/tree/5d450505 Diff: http://git-wip-us.apache.org/repos/asf/bahir/diff/5d450505 Branch: refs/heads/master Commit: 5d450505cd58c0e831a1a3932408748906853cf6 Parents: 898980d Author: Sean Owen <so...@cloudera.com> Authored: Fri Feb 19 10:26:38 2016 +0000 Committer: Sean Owen <so...@cloudera.com> Committed: Fri Feb 19 10:26:38 2016 +0000 ---------------------------------------------------------------------- streaming-mqtt/python/dstream.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/bahir/blob/5d450505/streaming-mqtt/python/dstream.py ---------------------------------------------------------------------- diff --git a/streaming-mqtt/python/dstream.py b/streaming-mqtt/python/dstream.py index 86447f5..2056663 100644 --- a/streaming-mqtt/python/dstream.py +++ b/streaming-mqtt/python/dstream.py @@ -453,7 +453,7 @@ class DStream(object): 2. "inverse reduce" the old values that left the window (e.g., subtracting old counts) This is more efficient than `invReduceFunc` is None. - @param reduceFunc: associative reduce function + @param reduceFunc: associative and commutative reduce function @param invReduceFunc: inverse reduce function of `reduceFunc` @param windowDuration: width of the window; must be a multiple of this DStream's batching interval @@ -524,7 +524,7 @@ class DStream(object): `invFunc` can be None, then it will reduce all the RDDs in window, could be slower than having `invFunc`. - @param func: associative reduce function + @param func: associative and commutative reduce function @param invFunc: inverse function of `reduceFunc` @param windowDuration: width of the window; must be a multiple of this DStream's batching interval