So the summarize (I had a similar question): Spark's log4j per default is configured to log to the console? Those messages end up in the stderr files and the approach does not support rolling?
If I configure log4j to log to files, how can I keep the folder structure? Should I use relative paths and assume that those end up in the same folders the stderr files do? Regards, Jeff 2015-02-25 9:35 GMT+01:00 Sean Owen <so...@cloudera.com>: > These settings don't control what happens to stderr, right? stderr is > up to the process that invoked the driver to control. You may wish to > configure log4j to log to files instead. > > On Wed, Nov 12, 2014 at 8:15 PM, Nguyen, Duc <duc.ngu...@pearson.com> > wrote: > > I've also tried setting the aforementioned properties using > > System.setProperty() as well as on the command line while submitting the > job > > using --conf key=value. All to no success. When I go to the Spark UI and > > click on that particular streaming job and then the "Environment" tab, I > can > > see the properties are correctly set. But regardless of what I've tried, > the > > "stderr" log file on the worker nodes does not roll and continues to > > grow...leading to a crash of the cluster once it claims 100% of disk. Has > > anyone else encountered this? Anyone? > > > > > > > > On Fri, Nov 7, 2014 at 3:35 PM, Nguyen, Duc <duc.ngu...@pearson.com> > wrote: > >> > >> We are running spark streaming jobs (version 1.1.0). After a sufficient > >> amount of time, the stderr file grows until the disk is full at 100% and > >> crashes the cluster. I've read this > >> > >> https://github.com/apache/spark/pull/895 > >> > >> and also read this > >> > >> http://spark.apache.org/docs/latest/configuration.html#spark-streaming > >> > >> > >> So I've tried testing with this in an attempt to get the stderr log file > >> to roll. > >> > >> sparkConf.set("spark.executor.logs.rolling.strategy", "size") > >> .set("spark.executor.logs.rolling.size.maxBytes", "1024") > >> .set("spark.executor.logs.rolling.maxRetainedFiles", "3") > >> > >> > >> Yet it does not roll and continues to grow. Am I missing something > >> obvious? > >> > >> > >> thanks, > >> Duc > >> > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
