Hi All, I have a system with 6 physical cores and each core has 8 hardware threads resulting in 48 virtual cores. Following are the setting in configuration files.
*spark-env.sh* export SPARK_WORKER_CORES=1 *spark-defaults.conf* spark.driver.cores 1 spark.executor.cores 1 spark.cores.max 1 So it means it should only use 1 virtual core but if we see the output from the TOP command, some time, it has very huge spikes e.g the CPU consumption is above 4000 e.g. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22581 sbaig 20 0 0.278t 0.064t 37312 S 4728 6.4 7:11.30 java .... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22581 sbaig 20 0 0.278t 0.065t 37312 S 1502 6.5 8:22.75 java ... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22581 sbaig 20 0 0.278t 0.065t 37312 S 4035 6.6 9:51.64 java ... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22581 sbaig 20 0 0.278t 0.080t 37312 S 3445 8.1 15:06.26 java ... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22581 sbaig 20 0 0.278t 0.082t 37312 S 4178 8.2 17:37.59 java ... It means, instead of using 1 virtual core, spark is using all available cores in the system so my question is why it is behaving like this? why it is not using only 1 core during execution of job which we set in SPARK_WORKER_CORES property. I am using spark 1.6.1 with standalone mode. Any help will be highly appreciated. Thanks Shuja -- Regards Shuja-ur-Rehman Baig <http://pk.linkedin.com/in/shujamughal>