[ https://issues.apache.org/jira/browse/SPARK-29553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977236#comment-16977236 ]
WuZeyi commented on SPARK-29553: -------------------------------- Thanks for Looking. I modify spark-env.sh,and the flame graph is like this: !image-2019-11-19-16-11-43-130.png! Then I modify {color:#172b4d}spark.conf, and the flame graph is this:{color} {color:#172b4d}!image-2019-11-19-16-13-30-723.png!{color} It means that it is still multi-thread if I modify spark-env.sh. If I modify spark.conf to set {color:#ff0000}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color},it works and the the performance improve. IMHO, spark-env,sh only set the evn of the spark-sumbit process, but it doesn't work in the executor processes. > This problemis about using native BLAS to improvement ML/MLLIB performance > -------------------------------------------------------------------------- > > Key: SPARK-29553 > URL: https://issues.apache.org/jira/browse/SPARK-29553 > Project: Spark > Issue Type: Improvement > Components: ML, MLlib > Affects Versions: 2.3.0, 2.4.4 > Reporter: WuZeyi > Priority: Minor > Labels: performance > Attachments: image-2019-11-19-16-11-43-130.png, > image-2019-11-19-16-13-30-723.png > > > I use {color:#ff0000}native BLAS{color} to improvement ML/MLLIB performance > on Yarn. > The file {color:#ff0000}spark-env.sh{color} which is modified by SPARK-21305 > said that I should set {color:#ff0000}OPENBLAS_NUM_THREADS=1{color} to > disable multi-threading of OpenBLAS, but it does not take effect. > I modify {color:#ff0000}spark.conf{color} to set > {color:#FF0000}spark.executorEnv.OPENBLAS_NUM_THREADS=1{color},and the > performance improve. > > > I think MKL_NUM_THREADS is the same. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org