convert List to RDD
Hi, I have a List[ (String, Int, Int) ] that I would liek to convert to an RDD. I tried to use sc.parallelize and sc.makeRDD, but in each case the original order of items in the List gets modified. Is there a simple way to convert a List to RDD without using SparkContext? thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/convert-List-to-RDD-tp7606.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: convert List to RDD
I may be wrong, but I think RDDs must be created inside a SparkContext. To somehow preserve the order of the list, perhaps you could try something like: sc.parallelize((1 to xs.size).zip(xs)) On Fri, Jun 13, 2014 at 6:08 PM, SK wrote: > Hi, > > I have a List[ (String, Int, Int) ] that I would liek to convert to an RDD. > I tried to use sc.parallelize and sc.makeRDD, but in each case the original > order of items in the List gets modified. Is there a simple way to convert a > List to RDD without using SparkContext? > > thanks > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/convert-List-to-RDD-tp7606.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: convert List to RDD
Thanks. But that did not work. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/convert-List-to-RDD-tp7606p7609.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: convert List to RDD
Sorry I wasn't being clear. The idea off the top of my head was that you could append an original position index to each element (using the line above), and modified what ever processing functions you have in mind to make them aware of these indices. And I think you are right that RDD collections are unordered by default. On Fri, Jun 13, 2014 at 6:33 PM, SK wrote: > Thanks. But that did not work. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/convert-List-to-RDD-tp7606p7609.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.