So the summarize (I had a similar question):
Spark's log4j per default is configured to log to the console? Those
messages end up in the stderr files and the approach does not support
rolling?
If I configure log4j to log to files, how can I keep the folder structure?
Should I use relative paths
I think that's up to you. You can make it log wherever you want, and
have some control over how log4j names the rolled log files by
configuring its file-based rolling appender.
On Thu, Feb 26, 2015 at 10:05 AM, Jeffrey Jedele
jeffrey.jed...@gmail.com wrote:
So the summarize (I had a similar
These settings don't control what happens to stderr, right? stderr is
up to the process that invoked the driver to control. You may wish to
configure log4j to log to files instead.
On Wed, Nov 12, 2014 at 8:15 PM, Nguyen, Duc duc.ngu...@pearson.com wrote:
I've also tried setting the
I'm also facing the same issue.
I tried the configurations but it seems the executors spark's
log4j.properties seems to override the passed values, so you have to change
/etc/spark/conf/log4j.properties.
Let me know if any of you have managed to get this fixes programatically.
I am planning to
I've also tried setting the aforementioned properties using
System.setProperty() as well as on the command line while submitting the
job using --conf key=value. All to no success. When I go to the Spark UI
and click on that particular streaming job and then the Environment tab,
I can see the
We are running spark streaming jobs (version 1.1.0). After a sufficient
amount of time, the stderr file grows until the disk is full at 100% and
crashes the cluster. I've read this
https://github.com/apache/spark/pull/895
and also read this
