Re: Location preferences in pyspark?
Thanks, we decided to try to add the support ourselves :). On Tue, Oct 20, 2015 at 6:40 PM, Jeff Zhang wrote: > Yes, I don't think there is. You can use SparkContext.parallelize() to > make a RDD from a list. But no location preferences supported yet. > > On Sat, Oct 17, 2015 at 8:42 AM, Philip Weaver > wrote: > >> I believe what I want is the exact functionality provided by >> SparkContext.makeRDD in Scala. For each element in the RDD, I want specify >> a list of preferred hosts for processing that element. >> >> It looks like this method only exists in Scala, and as far as I can tell >> there is no similar functionality available in python. Is this true? >> >> - Philip >> >> > > > -- > Best Regards > > Jeff Zhang >
Re: Location preferences in pyspark?
Yes, I don't think there is. You can use SparkContext.parallelize() to make a RDD from a list. But no location preferences supported yet. On Sat, Oct 17, 2015 at 8:42 AM, Philip Weaver wrote: > I believe what I want is the exact functionality provided by > SparkContext.makeRDD in Scala. For each element in the RDD, I want specify > a list of preferred hosts for processing that element. > > It looks like this method only exists in Scala, and as far as I can tell > there is no similar functionality available in python. Is this true? > > - Philip > > -- Best Regards Jeff Zhang
Location preferences in pyspark?
I believe what I want is the exact functionality provided by SparkContext.makeRDD in Scala. For each element in the RDD, I want specify a list of preferred hosts for processing that element. It looks like this method only exists in Scala, and as far as I can tell there is no similar functionality available in python. Is this true? - Philip
