Apache Spark Log4j logging applicationId

Hi, I would like to add the applicationId to all logs produced by Spark through Log4j. Consider that I have a cluster with several jobs running in it, so the presence of the applicationId would be useful to logically divide them. I have found a partial solution. If I change the layout of the

Spark event logging with s3a

We are trying to use spark event logging with s3a as a destination for event data. We added these settings to the spark submits: spark.eventLog.dir s3a://ourbucket/sparkHistoryServer/eventLogs spark.eventLog.enabled true Everything works fine with smaller jobs, and we can see the history data

Spark Streaming logging on Yarn : issue with rolling in yarn-client mode for driver log

Hi All, I am running my spark streaming in yarn-client mode. I want to enable rolling and aggregation in node manager container. I am using configs as suggested in spark doc :

Spark and logging

I'm wondering how logging works in Spark. I see that there's the log4j.properties.template file in the conf directory. Safe to assume Spark is using log4j 1? What's the approach if we're using log4j 2? I've got a log4j2.xml file in the job jar which seems to be working for my log statements

Re: Spark and logging

only an answer to one of your questions: What about log statements in the partition processing functions? Will their log statements get logged into a file residing on a given 'slave' machine, or will Spark capture this log output and divert it into the log file of the driver's machine?