When numPartitions is 0, there is no data in the rdd: so getPartition is
never invoked.
- Mridul
On Friday, September 16, 2016, WangJianfei
wrote:
> if so, we will get exception when the numPartitions is 0.
> def getPartition(key: Any): Int = key match {
> case null => 0
> //case None
if so, we will get exception when the numPartitions is 0.
def getPartition(key: Any): Int = key match {
case null => 0
//case None => 0
case _ => Utils.nonNegativeMod(key.hashCode, numPartitions)
}
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.na
They are valid, especially in partition pruning.
On Friday, September 16, 2016, Sean Owen wrote:
> There are almost no cases in which you'd want a zero-partition RDD.
> The only one I can think of is an empty RDD, where the number of
> partitions is irrelevant. Still, I would not be surprised if
There are almost no cases in which you'd want a zero-partition RDD.
The only one I can think of is an empty RDD, where the number of
partitions is irrelevant. Still, I would not be surprised if other
parts of the code assume at least 1 partition.
Maybe this check could be tightened. It would be in
class HashPartitioner(partitions: Int) extends Partitioner {
require(partitions >= 0, s"Number of partitions ($partitions) cannot be
negative.")
the soruce code require(partitions >=0) ,but I don't know why it makes sense
when the partitions is 0.
--
View this message in context:
http://apac
