Hi Rob,
I fear your questions will be hard to answer without additional information
about what kind of simulations you plan to do. int[r][c] basically means
you have a matrix of integers? You could for example map this to a
row-oriented RDD of integer-arrays or to a column oriented RDD of integer
arrays. What the better option is will heavily depend on your workload.
Also have a look at the algebaraic data-structures that come with mllib (
https://spark.apache.org/docs/1.1.0/api/scala/index.html#org.apache.spark.mllib.linalg.Vectors
).
Regards,
Jeff
2015-02-25 23:58 GMT+01:00 Rob Sargent rob.sarg...@utah.edu:
I have an application which might benefit from Sparks
distribution/analysis, but I'm worried about the size and structure of my
data set. I need to perform several thousand simulation on a rather large
data set and I need access to all the generated simulations. The data
element is largely in int[r][c] where r is 100 to 1000 and c is 20-80K
(there's more but that array is the bulk of the problem. I have machines
and memory capable of doing 6-10 simulations simultaneously in separate
jvms. Is this data structure compatible with Sparks RDD notion?
If yes, I will have a slough of how-to-get-started questions, the first of
which is how to seed the run? My thinking is to use
org.apache.spark.api.java.FlatMapFunction starting with an EmptyRDD and
the seed data. Would that be the way to go?
Thanks
