Hi, In case of MR task the log4j configuration and container log folder for a container is explicitly set in the container Launch context by "org.apache.hadoop.mapreduce.v2.util.MRApps.addLog4jSystemProperties" i.e from MapReduce YARN client code and not YARN code. This is also visible from jinfo on the MR's container's pid "-Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=<container log dir>/application_1413959638984_0032/container_1413959638984_0032_01_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m"
But in case of spark we don't see the log4j.configuration=container-log4j.propertiesto be set by default then -XX:OnOutOfMemoryError=kill %p -Xms512m -Xmx512m -Djava.io.tmpdir=<local dir>/usercache/root/appcache/application_1413959638984_0027/container_1413959638984_0027_01_000003/tmp -Dlog4j.configuration.watch=false -Dspark.akka.timeout=100 -Dspark.akka.frameSize=10 -Dspark.akka.heartbeat.pauses=600 -Dspark.akka.threads=4 -Dspark.yarn.app.container.log.dir=<container log dir>/application_1413959638984_0027/container_1413959638984_0027_01_000003 If the user doesn't set custom log4j configuration, how default spark container log4j settings are done ? Is any Log4j configuration file is set in the class path/jars or the log4j appender is programmatically set ? As the container log folder is different for each container, static log4j cannot be used so is spark.yarn.app.container.log.dir might be getting set in the log4j.properties but not clear who or where its set, can anyone give more information on this ? Regards, Ramana --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
