[
https://issues.apache.org/jira/browse/SPARK-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073310#comment-14073310
]
Thomas Graves commented on SPARK-2668:
--------------------------------------
Oh, I see you just want a variable to reference from the log4j config. I
understand the use case and really YARN should solve this for you. There are
jira out there to support long running tasks on yarn, the one for logs is:
https://issues.apache.org/jira/browse/YARN-1104
This might be ok for short term workaround for that since its just reading and
not allowing user to set it. I need to look at it a bit closer.
> Support log4j log to yarn container log directory
> -------------------------------------------------
>
> Key: SPARK-2668
> URL: https://issues.apache.org/jira/browse/SPARK-2668
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Reporter: Peng Zhang
> Fix For: 1.0.0
>
>
> Assign value of yarn container log directory to java opts
> "spark.yarn.log.dir", So user defined log4j.properties can reference this
> value and write log to YARN container directory.
> Otherwise, user defined file append will log to CWD, and files will not be
> displayed on YARN UIļ¼and either cannot be aggregated to HDFS log directory
> after job finished.
> User defined log4j.properties reference example:
> {code}
> log4j.appender.rolling_file.File = ${spark.yarn.log.dir}/spark.log
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
