HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458386348 Looks like in unmanaged mode some attributes are not available. ``` 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: 19/01/29 10:55:03 INFO YarnContainerInfoHelper: Error while building executor logs - executor logs will not be available 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: java.lang.NumberFormatException: null 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at java.lang.Integer.parseInt(Integer.java:542) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at java.lang.Integer.parseInt(Integer.java:615) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringLike.toInt(StringLike.scala:304) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringLike.toInt$(StringLike.scala:304) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringOps.toInt(StringOps.scala:33) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.util.YarnContainerInfoHelper$.getNodeManagerHttpPort(YarnContainerInfoHelper.scala:105) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.util.YarnContainerInfoHelper$.getLogUrls(YarnContainerInfoHelper.scala:41) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.$anonfun$prepareEnvironment$4(ExecutorRunnable.scala:240) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.$anonfun$prepareEnvironment$4$adapted(ExecutorRunnable.scala:239) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at scala.Option.foreach(Option.scala:274) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.prepareEnvironment(ExecutorRunnable.scala:239) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:89) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:66) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.YarnAllocator$$anon$1.run(YarnAllocator.scala:532) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at java.lang.Thread.run(Thread.java:748) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: 19/01/29 10:55:03 INFO YarnContainerInfoHelper: Error while retrieving executor attributes - executor logs will not be replaced with custom log pattern 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: java.lang.NumberFormatException: null 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at java.lang.Integer.parseInt(Integer.java:542) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at java.lang.Integer.parseInt(Integer.java:615) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringLike.toInt(StringLike.scala:304) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringLike.toInt$(StringLike.scala:304) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringOps.toInt(StringOps.scala:33) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.util.YarnContainerInfoHelper$.getNodeManagerPort(YarnContainerInfoHelper.scala:103) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.util.YarnContainerInfoHelper$.getAttributes(YarnContainerInfoHelper.scala:64) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.$anonfun$prepareEnvironment$4(ExecutorRunnable.scala:246) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.$anonfun$prepareEnvironment$4$adapted(ExecutorRunnable.scala:239) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at scala.Option.foreach(Option.scala:274) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.prepareEnvironment(ExecutorRunnable.scala:239) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:89) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:66) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.YarnAllocator$$anon$1.run(YarnAllocator.scala:532) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at java.lang.Thread.run(Thread.java:748) ``` First one: ``` org.apache.spark.util.YarnContainerInfoHelper$.getNodeManagerHttpPort(YarnContainerInfoHelper.scala:105) ``` Second one: ``` 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.util.YarnContainerInfoHelper$.getNodeManagerPort(YarnContainerInfoHelper.scala:103) ``` which will make attribute Map being empty and fail the test. Please note that it sometimes works, not always fails. Is it expected behavior for unmanaged mode to not have NM http port as well as NM port in system env? If then we can change the test to not check attributes. If not we need to investigate it.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
