GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/20414
[SPARK-23243][SQL] Shuffle+Repartition on an RDD could lead to incorrect
answers
## What changes were proposed in this pull request?
The RDD repartition also uses the round-robin way to distribute data, this
can also cause incorrect answers on RDD workload the similar way as in #20393
However, the approach that fixes DataFrame.repartition() doesn't apply on
the RDD repartition issue, because the input data can be non-comparable, as
discussed in https://github.com/apache/spark/pull/20393#issuecomment-360912451
Here, I propose a quick fix that distribute elements use their hashes, this
will cause perf regression if you have highly skewed input data, but it will
ensure result correctness.
## How was this patch tested?
Added test case in `RDDSuite` to ensure `RDD.repartition()` generate
consistent answers.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jiangxb1987/spark rdd-repartition
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20414.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20414
commit 6910ed62c272bedfa251cab589bb52bed36be3ed
Author: Xingbo Jiang
Date: 2018-01-27T00:34:24Z
fix RDD.repartition()
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
