Hello, I am trying to setup distributed cluster with separate all services but i have problem while running train:
log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /pio/pio.log (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:133) at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.apache.spark.internal.Logging$class.initializeLogging(Logging.scala:117) at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:102) at org.apache.spark.deploy.yarn.ApplicationMaster$.initializeLogIfNecessary(ApplicationMaster.scala:738) at org.apache.spark.internal.Logging$class.log(Logging.scala:46) at org.apache.spark.deploy.yarn.ApplicationMaster$.log(ApplicationMaster.scala:738) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:753) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) setup: hbase Hadoop Hdfs Spark cluster with yarn Training in cluster mode I assume spark worker is trying to save log to /pio/pio.log on worker machine instead of pio host. How can I set pio destination to hdfs path ? Or any other advice ? Thanks, Wojciech