No it's running in standalone mode as Docker image on Kubernetes.
The only way I found was to access "stderr" file created under the "work" directory in the SPARK_HOME but ... is it the right way ? Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoT Microsoft Azure Advisor Twitter : @ppatierno<http://twitter.com/ppatierno> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> Blog : DevExperience<http://paolopatierno.wordpress.com/> ________________________________ From: Alonso Isidoro Roman <alons...@gmail.com> Sent: Wednesday, May 31, 2017 8:39 AM To: Paolo Patierno Cc: user@spark.apache.org Subject: Re: Worker node log not showed Are you running the code with yarn? if so, figure out the applicationID through the web ui, then run the next command: yarn logs your_application_id <https://about.me/alonso.isidoro.roman?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links> Alonso Isidoro Roman about.me/alonso.isidoro.roman 2017-05-31 9:42 GMT+02:00 Paolo Patierno <ppatie...@live.com<mailto:ppatie...@live.com>>: Hi all, I have a simple cluster with one master and one worker. On another machine I launch the driver where at some point I have following line of codes : max.foreachRDD(rdd -> { LOG.info("*** max.foreachRDD"); rdd.foreach(value -> { LOG.info("*** rdd.foreach"); }); }); The message "*** max.foreachRDD" is visible in the console of the driver machine ... and it's ok. I can't see the "*** rdd.foreach" message that should be executed on the worker node right ? Btw on the worker node console I can't see it. Why ? My need is to log what happens in the code executed on worker node (because it works if I execute it on master local[*] but not when submitted to a worker node). Following the log4j.properties file I put in the /conf dir : # Set everything to be logged to the console log4j.rootCategory=INFO, console log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.err log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n # Settings to quiet third party logs that are too verbose log4j.logger.org.spark-project.jetty=WARN log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO Thanks Paolo. Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoT Microsoft Azure Advisor Twitter : @ppatierno<http://twitter.com/ppatierno> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> Blog : DevExperience<http://paolopatierno.wordpress.com/>
