[ https://issues.apache.org/jira/browse/SPARK-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960922#comment-15960922 ]
Torsten Scholak commented on SPARK-16784: ----------------------------------------- I having this exact problem. I need to be able to change the log settings depending on the job and/or the application. The method illustrated above, i.e. specifying spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties, spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties using spark-submit with "--files log4j.properties" does indeed NOT work. However, I was surprised to find it suggested as solution on stackoverflow (http://stackoverflow.com/questions/28454080/how-to-log-using-log4j-to-local-file-system-inside-a-spark-application-that-runs), since it is in direct contradiction to issue described in this ticket. [~jbacon] have you created a follow-up ticket? > Configurable log4j settings > --------------------------- > > Key: SPARK-16784 > URL: https://issues.apache.org/jira/browse/SPARK-16784 > Project: Spark > Issue Type: Improvement > Affects Versions: 2.0.0, 2.1.0 > Reporter: Michael Gummelt > > I often want to change the logging configuration on a single spark job. This > is easy in client mode. I just modify log4j.properties. It's difficult in > cluster mode, because I need to modify the log4j.properties in the > distribution in which the driver runs. I'd like a way of setting this > dynamically, such as a java system property. Some brief searching showed > that log4j doesn't seem to accept such a property, but I'd like to open up > this idea for further comment. Maybe we can find a solution. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org