[
https://issues.apache.org/jira/browse/SPARK-8048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601492#comment-14601492
]
Madhusudanan Kandasamy commented on SPARK-8048:
-----------------------------------------------
In few cases it is required so we cannot make it to fail, for example -
class RangePartitioner[K : Ordering : ClassTag, V](
@transient partitions: Int,
@transient rdd: RDD[_ <: Product2[K,V]],
private var ascending: Boolean = true)
extends Partitioner {
// We allow partitions = 0, which happens when sorting an empty RDD under the
default settings.
require(partitions >= 0, s"Number of partitions cannot be negative but found
$partitions.")
> Explicit partitionning of an RDD with 0 partition will yield empty outer join
> -----------------------------------------------------------------------------
>
> Key: SPARK-8048
> URL: https://issues.apache.org/jira/browse/SPARK-8048
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.1
> Reporter: Olivier Toupin
> Priority: Minor
>
> Check this code =>
> https://gist.github.com/anonymous/0f935915f2bc182841f0
> Because of this => {{.partitionBy(new HashPartitioner(0))}}
> The join will return empty result.
> Here a normal expected behaviour would the join to crash, cause error, or to
> return unjoined results, but instead will yield an empty RDD.
> This a trivial exemple, but imagine:
> {{.partitionBy(new HashPartitioner(previous.partitions.length))}}.
> You join on an empty "previous" rdd, the lookup table is empty, Spark will
> you lose all your results, instead of returning unjoined results, and this
> without warnings or errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
