Re: How to create RDD from Java in-memory data?

2014-03-11 Thread wallacemann
Ah! Thank you. That'll work for now. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-create-RDD-from-Java-in-memory-data-tp2486p2570.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: How to create RDD from Java in-memory data?

2014-03-11 Thread wallacemann
In a similar vein, it would be helpful to have an Iterable way to access the data inside an RDD. The collect method takes everything in the RDD and puts in a list, but this blows up memory. Since everything I want is already inside the RDD, it could be easy to iterate over the content without

How to create RDD from Java in-memory data?

2014-03-10 Thread wallacemann
I would like to construct an RDD from data I already have in memory as POJO objects. Is this possible? For example, is it possible to create an RDD from IterableString? I'm running Spark from Java as a stand-alone application. The JavaWordCount example runs fine. In the example, the initial

Re: How to create RDD from Java in-memory data?

2014-03-10 Thread wallacemann
I was right ... I was missing something obvious. The answer to my question is to use JavaSparkContext.parallelize which works with ListT or ListTuple2lt;K,V. -- View this message in context: