[
https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073278#comment-14073278
]
Peng Zhang commented on SPARK-2668:
-----------------------------------
[~tgraves] Original log works fine, and log will be written to yarn container
log directory and named as "stderr".
But when I want to define my own log4j configuration, for example using
RollingAppender to avoid log file too big, especially for spark Streaming(7 x
24 hours), I should can't specify the base directory for log.
So adding "spark.yarn.log.dir" will help for reference in log4j.properties,
like the example in description.
Otherwise, log files will be located in container's working directory.
> Support log4j log to yarn container log directory
> -------------------------------------------------
>
> Key: SPARK-2668
> URL: https://issues.apache.org/jira/browse/SPARK-2668
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Reporter: Peng Zhang
> Fix For: 1.0.0
>
>
> Assign value of yarn container log directory to java opts
> "spark.yarn.log.dir", So user defined log4j.properties can reference this
> value and write log to YARN container directory.
> Otherwise, user defined file append will log to CWD, and files will not be
> displayed on YARN UIļ¼and either cannot be aggregated to HDFS log directory
> after job finished.
> User defined log4j.properties reference example:
> {code}
> log4j.appender.rolling_file.File = ${spark.yarn.log.dir}/spark.log
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
