[jira] [Resolved] (SPARK-8405) Show executor logs on Web UI when Yarn log aggregation is enabled
[ https://issues.apache.org/jira/browse/SPARK-8405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-8405. -- Resolution: Fixed Fix Version/s: 1.4.2 > Show executor logs on Web UI when Yarn log aggregation is enabled > - > > Key: SPARK-8405 > URL: https://issues.apache.org/jira/browse/SPARK-8405 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.4.0 >Reporter: Carson Wang >Assignee: Carson Wang > Fix For: 1.4.2 > > Attachments: SparkLogError.png > > > When running Spark application in Yarn mode and Yarn log aggregation is > enabled, customer is not able to view executor logs on the history server Web > UI. The only way for customer to view the logs is through the Yarn command > "yarn logs -applicationId ". > An screenshot of the error is attached. When you click an executor’s log link > on the Spark history server, you’ll see the error if Yarn log aggregation is > enabled. The log URL redirects user to the node manager’s UI. This works if > the logs are located on that node. But since log aggregation is enabled, the > local logs are deleted once log aggregation is completed. > The logs should be available through the web UIs just like other Hadoop > components like MapReduce. For security reasons, end users may not be able to > log into the nodes and run the yarn logs -applicationId command. The web UIs > can be viewable and exposed through the firewall if necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-8405) Show executor logs on Web UI when Yarn log aggregation is enabled
[ https://issues.apache.org/jira/browse/SPARK-8405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-8405. -- Resolution: Not A Problem > Show executor logs on Web UI when Yarn log aggregation is enabled > - > > Key: SPARK-8405 > URL: https://issues.apache.org/jira/browse/SPARK-8405 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.4.0 >Reporter: Carson Wang > Attachments: SparkLogError.png > > > When running Spark application in Yarn mode and Yarn log aggregation is > enabled, customer is not able to view executor logs on the history server Web > UI. The only way for customer to view the logs is through the Yarn command > "yarn logs -applicationId ". > An screenshot of the error is attached. When you click an executor’s log link > on the Spark history server, you’ll see the error if Yarn log aggregation is > enabled. The log URL redirects user to the node manager’s UI. This works if > the logs are located on that node. But since log aggregation is enabled, the > local logs are deleted once log aggregation is completed. > The logs should be available through the web UIs just like other Hadoop > components like MapReduce. For security reasons, end users may not be able to > log into the nodes and run the yarn logs -applicationId command. The web UIs > can be viewable and exposed through the firewall if necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
