Hello, I was wondering how Spark was enforcing to use *only* X number of cores per executor.
Is it simply running max Y tasks in parallel on each executor where X = Y * spark.task.cpus? (This is what I understood from browsing TaskSchedulerImpl). Which would mean the processing power used for"map"- (if any) and "reduce"-side shuffle sorting is unbound (ExternalAppendOnlyMap and ExternalSorter I guess)? Thanks, Thomas
