Re: spark 1.5.2 - value filterByRange is not a member of org.apache.spark.rdd.RDD[(myKey, myData)]

[Nirav Patel]

In second class I re-declared following and compile error went away. Your
soln worked too.

 implicit val rowKeyOrdering = rowKeyOrd

Thanks
Nirav



On Wed, Mar 30, 2016 at 7:36 PM, Ted Yu  wrote:

> Have you tried the following construct ?
>
> new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey()
>
> See core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala
>
> On Wed, Mar 30, 2016 at 5:20 AM, Nirav Patel 
> wrote:
>
>> Hi, I am trying to use filterByRange feature of spark OrderedRDDFunctions
>> in a hope that it will speed up filtering by scanning only required
>> partitions.
>> I have created Paired RDD with a RangePartitioner in one scala class and
>> in another class I am trying to access this RDD and do following:
>>
>> In first scala class called RDDInitializer  I do:
>>
>>  implicit val rowKeyOrdering = rowKeyOrd
>>
>> val repartitionRdd = rowdataRdd.partitionBy(new RangePartitioner(
>> minPartitions.toInt, dataRdd, true))
>>
>> dataRdd  = repartitionRdd.sortByKey()
>>
>>
>> In second scala class I do:
>>
>> import org.apache.spark.SparkContext._
>>
>> RDDInitializer.dataRdd.filterByRange(myKeyFirst, myKeyLast)
>> But I am getting following compile error:
>>
>> "value filterByRange is not a member of org.apache.spark.rdd.RDD[(myKey,
>> myData)]"
>>
>>
>> Looks like I can use all methods of OrderedRDDFunctions inside first
>> scala class where implicit rowKeyOrdering is defined but not in second
>> class.
>>
>>
>> Please help me resolve this compile error.
>>
>>
>> Thanks
>>
>> Nirav
>>
>>
>>
>>
>>
>> [image: What's New with Xactly] 
>>
>>   [image: LinkedIn]
>>   [image: Twitter]
>>   [image: Facebook]
>>   [image: YouTube]
>> 
>
>
>

-- 


[image: What's New with Xactly] 

  [image: LinkedIn] 
  [image: Twitter] 
  [image: Facebook] 
  [image: YouTube] 

Re: spark 1.5.2 - value filterByRange is not a member of org.apache.spark.rdd.RDD[(myKey, myData)]

[Ted Yu]

Have you tried the following construct ?

new OrderedRDDFunctions[K, V, (K, V)](rdd).sortByKey()

See core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala

On Wed, Mar 30, 2016 at 5:20 AM, Nirav Patel  wrote:

> Hi, I am trying to use filterByRange feature of spark OrderedRDDFunctions
> in a hope that it will speed up filtering by scanning only required
> partitions.
> I have created Paired RDD with a RangePartitioner in one scala class and
> in another class I am trying to access this RDD and do following:
>
> In first scala class called RDDInitializer  I do:
>
>  implicit val rowKeyOrdering = rowKeyOrd
>
> val repartitionRdd = rowdataRdd.partitionBy(new RangePartitioner(
> minPartitions.toInt, dataRdd, true))
>
> dataRdd  = repartitionRdd.sortByKey()
>
>
> In second scala class I do:
>
> import org.apache.spark.SparkContext._
>
> RDDInitializer.dataRdd.filterByRange(myKeyFirst, myKeyLast)
> But I am getting following compile error:
>
> "value filterByRange is not a member of org.apache.spark.rdd.RDD[(myKey,
> myData)]"
>
>
> Looks like I can use all methods of OrderedRDDFunctions inside first
> scala class where implicit rowKeyOrdering is defined but not in second
> class.
>
>
> Please help me resolve this compile error.
>
>
> Thanks
>
> Nirav
>
>
>
>
>
> [image: What's New with Xactly] 
>
>   [image: LinkedIn]
>   [image: Twitter]
>   [image: Facebook]
>   [image: YouTube]
> 

spark 1.5.2 - value filterByRange is not a member of org.apache.spark.rdd.RDD[(myKey, myData)]

[Nirav Patel]

Hi, I am trying to use filterByRange feature of spark OrderedRDDFunctions
in a hope that it will speed up filtering by scanning only required
partitions.
I have created Paired RDD with a RangePartitioner in one scala class and in
another class I am trying to access this RDD and do following:

In first scala class called RDDInitializer  I do:

 implicit val rowKeyOrdering = rowKeyOrd

val repartitionRdd = rowdataRdd.partitionBy(new RangePartitioner(
minPartitions.toInt, dataRdd, true))

dataRdd  = repartitionRdd.sortByKey()


In second scala class I do:

import org.apache.spark.SparkContext._

RDDInitializer.dataRdd.filterByRange(myKeyFirst, myKeyLast)
But I am getting following compile error:

"value filterByRange is not a member of org.apache.spark.rdd.RDD[(myKey,
myData)]"


Looks like I can use all methods of OrderedRDDFunctions inside first scala
class where implicit rowKeyOrdering is defined but not in second class.


Please help me resolve this compile error.


Thanks

Nirav

-- 


[image: What's New with Xactly] 

  [image: LinkedIn] 
  [image: Twitter] 
  [image: Facebook] 
  [image: YouTube]