Actually it doesn’t matter a lot from what I’ve seen. Only do it if you see a lot of communication going to the master (these threads do the serialization of tasks). I’ve never put more than 8 or so.
Matei On Nov 11, 2013, at 12:13 PM, Walrus theCat <walrusthe...@gmail.com> wrote: > Hi, > > The docs say that we should be careful to increase spark.akka.threads when > our cluster size increases. Is there a rule of thumb for how much we should > increase it? For instance, the recommendation for default parallelism is 2x > cores.
