shanyu zhao created SPARK-31028: ----------------------------------- Summary: Add "-XX:ActiveProcessorCount" to Spark driver and executor in Yarn mode Key: SPARK-31028 URL: https://issues.apache.org/jira/browse/SPARK-31028 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 2.4.5 Reporter: shanyu zhao
When starting Spark driver and executors on Yarn cluster, the JVM process can discover all CPU cores on the system and set thread-pool or GC threads based on that value. We should limit what the JVM sees for the number of cores set by the user (spark.driver.cores or spark.executor.cores) by "-XX:ActiveProcessorCount", which was introduced in Java 8u191. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org