Hi guys, any clue on this? Clearly the spark.executor.extraJavaOpts=-Dlog4j.configuration is not working on the executors.
Thanks, -carlos. On Wed, Apr 13, 2016 at 2:48 PM, Carlos Rojas Matas <cma...@despegar.com> wrote: > Hi Yong, > > thanks for your response. As I said in my first email, I've tried both the > reference to the classpath resource (env/dev/log4j-executor.properties) as > the file:// protocol. Also, the driver logging is working fine and I'm > using the same kind of reference. > > Below the content of my classpath: > > [image: Inline image 1] > > Plus this is the content of the exploded fat jar assembled with sbt > assembly plugin: > > [image: Inline image 2] > > > This folder is at the root level of the classpath. > > Thanks, > -carlos. > > On Wed, Apr 13, 2016 at 2:35 PM, Yong Zhang <java8...@hotmail.com> wrote: > >> Is the env/dev/log4j-executor.properties file within your jar file? Is >> the path matching with what you specified as >> env/dev/log4j-executor.properties? >> >> If you read the log4j document here: >> https://logging.apache.org/log4j/1.2/manual.html >> >> When you specify the log4j.configuration=my_custom.properties, you have 2 >> option: >> >> 1) the my_custom.properties has to be in the jar (or in the classpath). >> In your case, since you specify the package path, you need to make sure >> they are matched in your jar file >> 2) use like log4j.configuration=file:///tmp/my_custom.properties. In this >> way, you need to make sure file my_custom.properties exists in /tmp folder >> on ALL of your worker nodes. >> >> Yong >> >> ------------------------------ >> Date: Wed, 13 Apr 2016 14:18:24 -0300 >> Subject: Re: Logging in executors >> From: cma...@despegar.com >> To: yuzhih...@gmail.com >> CC: user@spark.apache.org >> >> >> Thanks for your response Ted. You're right, there was a typo. I changed >> it, now I'm executing: >> >> bin/spark-submit --master spark://localhost:7077 --conf >> "spark.driver.extraJavaOptions=-Dlog4j.configuration=env/dev/log4j-driver.properties" >> --conf >> "spark.executor.extraJavaOptions=-Dlog4j.configuration=env/dev/log4j-executor.properties" >> --class.... >> >> The content of this file is: >> >> # Set everything to be logged to the console >> log4j.rootCategory=INFO, FILE >> log4j.appender.console=org.apache.log4j.ConsoleAppender >> log4j.appender.console.target=System.err >> log4j.appender.console.layout=org.apache.log4j.PatternLayout >> log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p >> %c{1}: %m%n >> >> log4j.appender.FILE=org.apache.log4j.RollingFileAppender >> log4j.appender.FILE.File=/tmp/executor.log >> log4j.appender.FILE.ImmediateFlush=true >> log4j.appender.FILE.Threshold=debug >> log4j.appender.FILE.Append=true >> log4j.appender.FILE.MaxFileSize=100MB >> log4j.appender.FILE.MaxBackupIndex=5 >> log4j.appender.FILE.layout=org.apache.log4j.PatternLayout >> log4j.appender.FILE.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p >> %c{1}: %m%n >> >> # Settings to quiet third party logs that are too verbose >> log4j.logger.org.spark-project.jetty=WARN >> >> log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR >> log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO >> log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO >> log4j.logger.org.apache.parquet=ERROR >> log4j.logger.parquet=ERROR >> log4j.logger.com.despegar.p13n=DEBUG >> >> # SPARK-9183: Settings to avoid annoying messages when looking up >> nonexistent UDFs in SparkSQL with Hive support >> log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL >> log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR >> >> >> Finally, the code on which I'm using logging in the executor is: >> >> def groupAndCount(keys: DStream[(String, List[String])])(handler: >> ResultHandler) = { >> >> val result = keys.reduceByKey((prior, current) => { >> (prior ::: current) >> }).flatMap { >> case (date, keys) => >> val rs = keys.groupBy(x => x).map( >> obs =>{ >> val (d,t) = date.split("@") match { >> case Array(d,t) => (d,t) >> } >> import org.apache.log4j.Logger >> import scala.collection.JavaConverters._ >> val logger: Logger = Logger.getRootLogger >> logger.info(s"Metric retrieved $d") >> Metric("PV", d, obs._1, t, obs._2.size) >> } >> ) >> rs >> } >> >> result.foreachRDD((rdd: RDD[Metric], time: Time) => { >> handler(rdd, time) >> }) >> >> } >> >> >> Originally the import and logger object was outside the map function. I'm >> also using the root logger just to see if it's working, but nothing gets >> logged. I've checked that the property is set correctly on the executor >> side through println(System.getProperty("log4j.configuration")) and is OK, >> but still not working. >> >> Thanks again, >> -carlos. >> > >
