Hi all,
I have a simple cluster with one master and one worker. On another machine I
launch the driver where at some point I have following line of codes :
max.foreachRDD(rdd -> {
LOG.info("*** max.foreachRDD");
rdd.foreach(value -> {
LOG.info("*** rdd.foreach");
});
});
The message "*** max.foreachRDD" is visible in the console of the driver
machine ... and it's ok.
I can't see the "*** rdd.foreach" message that should be executed on the worker
node right ? Btw on the worker node console I can't see it. Why ?
My need is to log what happens in the code executed on worker node (because it
works if I execute it on master local[*] but not when submitted to a worker
node).
Following the log4j.properties file I put in the /conf dir :
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}:
%m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark-project.jetty=WARN
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
Thanks
Paolo.
Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor
Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>
