I have a cluster with 2 nodes (32 CPU cores each). My data is distributed evenly, but the processing times for each partition can vary greatly. Now, sometimes Spark seems to conclude from the current workload on both nodes that it might be better to shift one partition from node1 to node2 (because that guy has cores waiting for work). Am i hallucinating or is that really the happening? Is there any way I prevent this from happening?
Greetings, T3L -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Prevent-partitions-from-moving-tp25216.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
