Hi In am using an EMR cluster for running my spark jobs, but after the job finishes logs disappear,
I have added a log4j.properties in my jar, but all the logs still redirects to EMR resource manager which vanishes after jobs completes, is there a way i could redirect the logs to a location in file syatem, I am working on price points and its very critical for me to maintain logs. Just to add i get following error when my application starts. java.io.FileNotFoundException: /etc/spark/conf/log4j.properties (No such file or directory) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at java.io.FileInputStream.<init>(FileInputStream.java:101) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.apache.spark.Logging$class.initializeLogging(Logging.scala:122) at org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:107) at org.apache.spark.Logging$class.log(Logging.scala:51) at org.apache.spark.deploy.yarn.ApplicationMaster$.log(ApplicationMaster.scala:607) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:621) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)