This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch branch-3.3 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push: new 090bebd6a63 [SPARK-41254][YARN] bugfix wrong usage when check YarnAllocator.rpIdToYarnResource key existence 090bebd6a63 is described below commit 090bebd6a63fdd69b14d08c459fd5bd2301948e4 Author: John Caveman <selnt...@gmail.com> AuthorDate: Mon Nov 28 08:25:00 2022 -0600 [SPARK-41254][YARN] bugfix wrong usage when check YarnAllocator.rpIdToYarnResource key existence ### What changes were proposed in this pull request? bugfix, a misuse of ConcurrentHashMap.contains causing map YarnAllocator.rpIdToYarnResource always updated ### Why are the changes needed? It causing duplicated log during yarn resource allocation and unnecessary object creation and gc ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing tests Closes #38790 from CavemanIV/SPARK-41254. Authored-by: John Caveman <selnt...@gmail.com> Signed-off-by: Sean Owen <sro...@gmail.com> (cherry picked from commit bccfe5bca600b3091ea93b4c5d6437af8381973f) Signed-off-by: Sean Owen <sro...@gmail.com> --- .../src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala index a85b7174673..16cae4810e4 100644 --- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala +++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala @@ -278,7 +278,7 @@ private[yarn] class YarnAllocator( // if a ResourceProfile hasn't been seen yet, create the corresponding YARN Resource for it private def createYarnResourceForResourceProfile(rp: ResourceProfile): Unit = synchronized { - if (!rpIdToYarnResource.contains(rp.id)) { + if (!rpIdToYarnResource.containsKey(rp.id)) { // track the resource profile if not already there getOrUpdateRunningExecutorForRPId(rp.id) logInfo(s"Resource profile ${rp.id} doesn't exist, adding it") --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org