Is this the sort of problem spark can accommodate? I need to compare 10,000 matrices with each other (10^10 comparison). The matrices are 100x100000 (10^7 int values). I have 10 machines with 2 to 8 cores (8-32 "processors"). All machines have to - contribute to matrices generation (a simulation, takes seconds) - see all matrices - compare matrices (takes very little time compared to simulation)
I expect to persist the simulations, have spark push them to processors. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/distributing-large-matrices-tp24174.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
