[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-459109883 Thanks all for detailed reviewing and merging! I'll file a new issue and raise a PR soon for executor self-retrieving information. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458445037 Retest this, please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458444999 Two build failures don't look relevant to YARN tests. Will retrigger tests... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458445072 Retest this, please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458411180 Commit trigger and two manual requests : 3 builds. Let's see how it goes. I checked YarnClusterSuite passed in local devs 3 times in a row. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458411022 retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458410914 retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458410796 I just fixed a bug: the reason is we cannot retrieve NM_HOST / NM_PORT / NM_HTTP_PORT outside of container process for given container. In YARN cluster mode we can get them for driver, but not for executors in both cluster and client mode. In order to roll back these attributes to NM_HTTP_ADDRESS, I had to remove explanation on setting up custom log URL to JHS URL because it refers to NM_PORT (IPC) which cannot be retrieved. We might try moving out extracting log urls and attributes to YARN executor side and see whether it works, but I'd hope addressing that in other issue. I would like to get it done with stable shape first, and try out more. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458386991 And very oddly, looks like unmanaged mode is running in "yarn-client" mode, but the test only verifies map when deploy mode is "cluster". This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458386348 Looks like in unmanaged mode some attributes are not available. ``` 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: 19/01/29 10:55:03 INFO YarnContainerInfoHelper: Error while building executor logs - executor logs will not be available 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: java.lang.NumberFormatException: null 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at java.lang.Integer.parseInt(Integer.java:542) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at java.lang.Integer.parseInt(Integer.java:615) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringLike.toInt(StringLike.scala:304) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringLike.toInt$(StringLike.scala:304) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringOps.toInt(StringOps.scala:33) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.util.YarnContainerInfoHelper$.getNodeManagerHttpPort(YarnContainerInfoHelper.scala:105) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.util.YarnContainerInfoHelper$.getLogUrls(YarnContainerInfoHelper.scala:41) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.$anonfun$prepareEnvironment$4(ExecutorRunnable.scala:240) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.$anonfun$prepareEnvironment$4$adapted(ExecutorRunnable.scala:239) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at scala.Option.foreach(Option.scala:274) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.prepareEnvironment(ExecutorRunnable.scala:239) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:89) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:66) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.YarnAllocator$$anon$1.run(YarnAllocator.scala:532) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: at java.lang.Thread.run(Thread.java:748) 19/01/29 10:55:03.993 launcher-proc-3 INFO YarnClusterDriver: 19/01/29 10:55:03 INFO YarnContainerInfoHelper: Error while retrieving executor attributes - executor logs will not be replaced with custom log pattern 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: java.lang.NumberFormatException: null 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at java.lang.Integer.parseInt(Integer.java:542) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at java.lang.Integer.parseInt(Integer.java:615) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringLike.toInt(StringLike.scala:304) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringLike.toInt$(StringLike.scala:304) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at scala.collection.immutable.StringOps.toInt(StringOps.scala:33) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.util.YarnContainerInfoHelper$.getNodeManagerPort(YarnContainerInfoHelper.scala:103) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.util.YarnContainerInfoHelper$.getAttributes(YarnContainerInfoHelper.scala:64) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.$anonfun$prepareEnvironment$4(ExecutorRunnable.scala:246) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.$anonfun$prepareEnvironment$4$adapted(ExecutorRunnable.scala:239) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at scala.Option.foreach(Option.scala:274) 19/01/29 10:55:03.994 launcher-proc-3 INFO YarnClusterDriver: at org.apache.spark.deploy.yarn.ExecutorRunnable.prepareEnvironment(ExecutorRunnable.scala:239) 19/01
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458368442 I'm seeing flakiness of YARN tests after rebasing, specifically SPARK-22404 (#19616). More failures than succeeds. I'm triggering the build multiple times, as well as running test in local dev. We should read container logs to track down the issue but I couldn't do against Jenkins build so have to deal with local env. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458288375 Addressed. Still not clear how to get user - in #23659 it is not addressed yet - and the patch may overwrite it. Btw, if we merge this PR earlier than #23659 we can just close #23659 since the code becomes no longer valid. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-457909705 Retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-457493559 Retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply
custom log URL pattern for executor log URLs in SHS
URL: https://github.com/apache/spark/pull/23260#issuecomment-457021478
I reread the comment and realized it can be simply resolved when you
construct your custom log URL to just point to the URL which serves log file
listing. Once `{{FILE_NAME}}` pattern is not given in custom log URL, it
doesn't deal with `{{LOG_FILES}}` and serve the URL as "log".
https://github.com/apache/spark/blob/944b30a127b7855b7f59b7f743c351d14bc5d07a/core/src/main/scala/org/apache/spark/deploy/history/HistoryAppStatusStore.scala#L102-L109
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456974106 @tgravescs Great idea. `stdout` and `stderr` are what Spark is providing regardless of log4j configuration, and I agree it should be nice to expose access to other files as well. But also the PR already handles broader than I intended as of now, so filing a new issue and starting discussion from that issue would be ideal. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456562634 @vanzin kindly asking to review this again, given I have changed the approach which satisfies requirements I guess. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS
HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-455759736 retest this, please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
