Hi, I'm trying to run a spark job that uses multiple cpu cores per spark executor in a spark job. Specifically, it runs the gemm matrix multiply routine from each partition on a large matrix that cannot be distributed.
For test purpose, I have a machine with 8 cores running standalone spark. I started a spark context, setting "spark.task.cpus" to "8"; then I generated an RDD with 1 partition only so there will be one executor using all cores. The job is coded in Java, with JNI wrapper provided by fommil (netlib-java) and underlying BLAS implementation from OpenBLAS, and the machine I'm running is one local desktop with Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz (8 cores) When I run the test using a local spark as "local[8]", I can see the routine completes in about 200ms, and CPU utilization is near 100% for all cores. This is nearly identical performance to running the same code without spark straight from Java. When I run the test attaching to the standalone spark by setting master as "spark://****:7077, the same code takes about 12 seconds, and monitoring the cpu shows that only one thread is used at a time. This is also very close to the performance I get if I ran the routine in Java with only one core. I do not see any warning about failure to load native library, and if I collect a map of System.getenv(), I see that all the environment variables seems to be correct (OPENBLAS_NUM_THREADS=8, LD_LIBRARY_PATH includes the wrapper, etc..) I also tried to replace OpenBLAS with MKL, with MKL_NUM_THREADS=8 and MKL_DYNAMIC=false, but I got exactly same behaviour: local spark seems to use all cores, but standalone spark would not use all cores. I tried a lot of different settings on the native library's side but it seems weird that local spark was okay but not the standalone spark. Any help is greatly appreciated! Guang -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Native-libraries-using-only-one-core-in-standalone-spark-cluster-tp27795.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
