Hi, Ashic, Thank you very much for your reply!
The tasks I mention is a running Function that I implemented with Spark API and passed to each partition of a RDD. Within the Function I broadcast a big variable to be queried by each partition. So, When I am running on a 48 cores slave node. I have 48 partitions corresponding 48 tasks (or clousure) where each tasks get a broadcast value (I see this from the memory usage and the API doc). Is there a way to share the value with all 48 partitions of 48 tasks? best! huanglr From: Ashic Mahtab Date: 2015-07-10 17:02 To: huanglr; Apache Spark Subject: RE: Spark Broadcasting large dataset When you say tasks, do you mean different applications, or different tasks in the same application? If it's the same program, they should be able to share the broadcasted value. But given you're asking the question, I imagine they're separate. And in that case, afaik, the answer is no. You might look into putting the data into a fast store like Cassandra - that might help depending on your use case. Cheers, Ashic. Date: Fri, 10 Jul 2015 15:52:42 +0200 From: huan...@cebitec.uni-bielefeld.de To: user@spark.apache.org Subject: Spark Broadcasting large dataset Hey, Guys! I am using spark for NGS data application. In my case I have to broadcast a very big dataset to each task. However there are serveral tasks (say 48 tasks) running on cpus (also 48 cores) in the same node. These tasks, who run on the same node, could share the same dataset. But spark broadcast them 48 times (if I understand correctly). Is there a way to broadcast just one copy for each node and share by all tasks running on such nodes? Much appreciated! best! huanglr
